diff --git a/docs/contents.md b/docs/contents.md index f3848d61b..3e3b3d1ca 100644 --- a/docs/contents.md +++ b/docs/contents.md @@ -37,6 +37,22 @@ operator, user, and contributor references are easier to find. - [rfcs/0001-structured-command-blocks.md](rfcs/0001-structured-command-blocks.md): Proposed structured command blocks, shell-free argv templates, typed Jinja interpolation, stream routing, and pipeline semantics. +- [rfcs/0009-structured-command-working-directories.md][rfc-0009]: + Normative amendment adding capability-scoped per-command working directories + to RFC 0001. +- [rfcs/0010-runtime-bindings-and-secure-tempdirs.md][rfc-0010]: + Normative amendment adding bounded stdout environment capture, standard-error + pipelines, environment-selected working directories, and secure temporary + execution directories to RFC 0001. +- [rfcs/0002-repository-relative-includes.md][rfc-0002]: + Deterministic repository-relative manifest includes with cycle detection, + provenance, namespaces, and duplicate rejection. +- [rfcs/0003-versioned-local-bundles.md](rfcs/0003-versioned-local-bundles.md): + Versioned local bundles with typed parameters, explicit exports, SemVer + selection, canonical digests, and lock records. +- [rfcs/0004-digest-pinned-external-bundles.md][rfc-0004]: + Later digest-pinned external Git bundles with exact tag resolution, immutable + object provenance, bounded acquisition, caching, and offline verification. - [RFC 0005: Harden release integrity and admission](rfcs/0005-release-hardening.md): Proposed release-profile invariants, secret and dependency policy, and exact-commit release admission. @@ -52,6 +68,11 @@ operator, user, and contributor references are easier to find. self-consistency, health-signal ownership, and scheduled coverage-guided fuzzing. +[rfc-0009]: rfcs/0009-structured-command-working-directories.md +[rfc-0010]: rfcs/0010-runtime-bindings-and-secure-tempdirs.md +[rfc-0002]: rfcs/0002-repository-relative-includes.md +[rfc-0004]: rfcs/0004-digest-pinned-external-bundles.md + ## Decision records - [adr-001-replace-serde-yml-with-serde-saphyr.md](adr-001-replace-serde-yml-with-serde-saphyr.md): diff --git a/docs/netsuke-design.md b/docs/netsuke-design.md index 1fd7c84a4..bb8f909cc 100644 --- a/docs/netsuke-design.md +++ b/docs/netsuke-design.md @@ -702,6 +702,72 @@ explicit, structured, and self-documenting nature. | Target Build | my_program: main.o utils.o\\t$(CC) $^ -o $@ | { targets: { name: my_program, rule: link, sources: [main.o, utils.o] } | | Readability | Relies on cryptic automatic variables ($@, $\<, $^) and implicit pattern matching. | Uses explicit, descriptive keys (name, rule, sources) and standard YAML list/map syntax. | +### 2.8 Manifest composition and bundle resolution + +RFC 0003 defines deterministic resolution for versioned local bundles. The +following sequence shows how a manifest resolves, validates, and locks a +namespaced bundle. + +For screen readers: The manifest asks the resolver to resolve a bundle source +and version. The resolver enumerates sorted catalogue candidates, selects the +highest compatible SemVer, validates parameters and exports, obtains the +canonical content digest, verifies the lock, and returns namespaced exported +declarations. + +```mermaid +sequenceDiagram + participant Manifest + participant Resolver + participant Catalogue + participant Bundle + participant Lock + + Manifest->>Resolver: resolve bundle source and version + Resolver->>Catalogue: enumerate candidates in sorted order + Catalogue-->>Resolver: bundle descriptors + Resolver->>Resolver: select highest compatible SemVer + Resolver->>Bundle: validate parameters and exports + Bundle-->>Resolver: canonical content digest + Resolver->>Lock: verify selected version and digests + Lock-->>Resolver: lock status + Resolver-->>Manifest: namespaced exported declarations +``` + +Figure: Local bundle resolution and lock verification. + +RFC 0004 extends this composition flow to external Git bundles. It preserves +the same verification boundary while adding exact reference resolution, +provenance, and content-addressed caching. + +For screen readers: The manifest asks the Git resolver for an exact tag or +commit. The resolver fetches normalized Git references or a full commit, then +the verifier checks the subdirectory, metadata, SemVer, and canonical digest. +The verifier reads or publishes the verified content-addressed bundle, compares +tag, peeled-commit, and digest values with the lock, and returns a verified +bundle instance to the manifest. + +```mermaid +sequenceDiagram + participant Manifest + participant GitResolver + participant GitRemote + participant Cache + participant Verifier + participant Lock + + Manifest->>GitResolver: resolve exact tag or commit + GitResolver->>GitRemote: fetch normalized refs/tags/... or full commit + GitRemote-->>GitResolver: Git objects and tag provenance + GitResolver->>Verifier: verify subdir, metadata, SemVer, and canonical digest + Verifier->>Cache: read or publish content-addressed bundle + Cache-->>Verifier: verified content + Verifier->>Lock: compare tag, peeled commit, and digest + Lock-->>Verifier: lock verification result + Verifier-->>Manifest: verified bundle instance +``` + +Figure: External Git bundle verification and cache-backed lock resolution. + ## Section 3: Parsing and Deserialization Strategy Once the Jinja evaluation stage has produced a pure YAML string, the next @@ -2353,6 +2419,35 @@ remains responsible for invoking Ninja correctly and, most importantly, for ensuring that the commands it generates for Ninja to run are constructed securely. +### Structured command working directories + +RFC 0009 gives structured commands an explicit, capability-checked working +directory. The sequence below shows that resolution and validation happen +before the child process receives its `current_dir`. + +For screen readers: The compiler provides a process specification with a +working directory to the action runner. The runner asks the workspace +capability to resolve and validate that directory, receives a +capability-relative directory, spawns the child process with that directory as +its current directory, and receives the process result. + +```mermaid +sequenceDiagram + participant Compiler + participant ActionRunner + participant WorkspaceCapability + participant ChildProcess + + Compiler->>ActionRunner: provide ProcessSpec with cwd + ActionRunner->>WorkspaceCapability: resolve and validate cwd + WorkspaceCapability-->>ActionRunner: capability-relative directory + ActionRunner->>ChildProcess: spawn with current_dir(cwd) + ChildProcess-->>ActionRunner: process result +``` + +Figure: Capability-checked working-directory resolution for a structured +command. + ### 6.1 Invoking Ninja Netsuke uses Rust's standard library `std::process::Command` API to configure diff --git a/docs/rfcs/0002-repository-relative-includes.md b/docs/rfcs/0002-repository-relative-includes.md new file mode 100644 index 000000000..ba6e227d6 --- /dev/null +++ b/docs/rfcs/0002-repository-relative-includes.md @@ -0,0 +1,468 @@ +# RFC 0002: Repository-relative manifest includes + +## Preamble + +- **RFC number:** 0002 +- **Status:** Proposed +- **Created:** 2026-08-26 +- **Target:** Netsuke manifest composition +- **Depends on:** RFC 0001 only for shared provenance and diagnostics concepts + +## 1. Summary + +This RFC introduces local, repository-relative manifest includes with a fixed +resolution order, deterministic merge semantics, cycle detection, retained +source provenance, and no network access. + +The initial feature is deliberately conservative. Included files are static +manifest fragments. Include paths are literal paths rather than Jinja +expressions or glob patterns, every resolved file must remain inside the +workspace boundary, and duplicate declarations are errors unless a future RFC +adds an explicit override construct. + +The central rule is: + +> Composition order is explicit, stable, and incapable of silently replacing a +> declaration. + +A root manifest includes fragments in declaration order. Each fragment first +composes its own includes, then contributes its declarations. The including +file contributes last. This produces a deterministic depth-first post-order +without depending on directory iteration, hash-map order, or filesystem +metadata order. + +## 2. Problem + +Real Netsukefiles repeat the same local task groups: + +- Rust formatting, Clippy, rustdoc, testing, and Whitaker actions; +- Python formatting, linting, type checking, and tests; +- release, audit, spelling, and documentation gates; +- feature-lane matrices; and +- local helper rules shared by several subprojects. + +Jinja macros can reduce repetition inside one manifest, but they do not provide +source-file boundaries. Copying one large Netsukefile between repositories +creates drift; splitting commands into shell scripts loses graph metadata and +discoverable action descriptions. + +A generic YAML include is not enough. Netsuke must define: + +- which path anchors an include; +- whether a symlink may escape the repository; +- the order in which nested fragments compose; +- what happens when two files define the same name; +- when Jinja evaluates relative to composition; +- how source spans survive into diagnostics; +- whether `help targets` can inspect the graph without rendering recipes; and +- which content contributes to the generated graph hash. + +Without a normative contract, two implementations can produce different graphs +from the same files while both claiming to support includes. + +## 3. Goals and non-goals + +### 3.1 Goals + +This RFC aims to: + +- split one repository's manifest into reviewable local fragments; +- resolve every include relative to the file containing the include; +- keep all initial include access within the effective workspace boundary; +- define one deterministic nested-composition order; +- reject cycles and duplicate declaration identities before graph generation; +- preserve file and source-span provenance through expansion and diagnostics; +- make every included byte part of graph invalidation and reproducibility; +- keep metadata queries side-effect-free; and +- provide a foundation for versioned bundles without prematurely adding a + package manager. + +### 3.2 Non-goals + +This RFC does not: + +- fetch network resources; +- resolve Git repositories, commits, branches, or tags; +- select among semantic versions; +- parameterize reusable bundles; +- permit Jinja, `env()`, `glob()`, or command output in include paths; +- silently override duplicate variables, rules, targets, or actions; +- include files outside the effective workspace root; +- define remote trust or signature policy; or +- make included fragments independently executable entry points. + +RFC 0003 adds versioned local bundles. RFC 0004 defines the later, +digest-pinned external Git boundary. + +## 4. Terminology + +- **Root manifest:** The Netsukefile selected by the CLI. +- **Fragment:** A YAML document loaded only through `includes`. +- **Including file:** The file whose `includes` entry names another fragment. +- **Composition unit:** One parsed root manifest or fragment with retained + provenance. +- **Effective workspace root:** The workspace selected after CLI `-C` and + manifest selection rules. +- **Canonical include identity:** The capability-relative, symlink-resolved file + identity used for cycle and duplicate-load detection. +- **Composition order:** The total order in which declarations enter the + composed manifest. + +## 5. Manifest syntax + +A manifest or fragment may define a top-level `includes` sequence: + +```yaml +includes: + - path: build/netsuke/rust-quality.yaml + - path: build/netsuke/release.yaml +``` + +The compact scalar form is equivalent: + +```yaml +includes: + - build/netsuke/rust-quality.yaml + - build/netsuke/release.yaml +``` + +The initial mapping accepts exactly these keys: + +| Field | Type | Default | Meaning | +| ------ | ---------------- | -------- | --------------------------------------------- | +| `path` | non-empty string | required | Fragment path relative to the including file. | +| `as` | identifier | absent | Namespace the included declarations. | + +Table 1: Local include fields. + +Unknown keys are errors. `path` and `as` are literal YAML strings. Jinja +expressions and structural Jinja blocks are invalid in both fields. + +### 5.1 Namespaced includes + +An include may place exported declaration names beneath a namespace: + +```yaml +includes: + - path: build/netsuke/rust-quality.yaml + as: rust_quality +``` + +The namespace applies to named rules, targets, and actions. For example, `lint` +becomes `rust_quality::lint`. References inside the fragment are rewritten +within that namespace before cross-fragment resolution. + +Fragment variables appear beneath one mapping value named by the namespace: + +```jinja +{{ rust_quality.vars.clippy_flags }} +``` + +The exact user-facing variable projection may be refined during implementation, +but it must remain a typed namespace rather than concatenating strings into +flat variable names. + +An unnamespaced include contributes declarations directly and is therefore more +likely to encounter duplicate-name errors. + +## 6. Resolution algorithm + +For each composition unit, Netsuke performs these steps: + +1. Initialize a composition-wide `visited` map keyed by canonical include + identity. The map retains the first include chain that loaded each identity. +2. Parse YAML sufficiently to identify `includes` without rendering Jinja. +3. For each include in declaration order: + 1. resolve `path` against the including file's parent directory; + 2. open the path through the workspace directory capability; + 3. resolve symlinks and obtain its canonical include identity; + 4. reject paths outside the effective workspace boundary; + 5. reject a canonical identity already active on the recursion stack; + 6. reject a canonical identity already present in `visited`, reporting the + first include chain and the attempted include chain; + 7. add the identity and its current chain to `visited`; + 8. parse the fragment; + 9. recursively compose that fragment's includes; and + 10. append the fragment's declarations to the composition stream. +4. Append the including file's own declarations. +5. Validate declaration identities and references over the complete stream. +6. Evaluate manifest-time Jinja and expand `foreach` and `when` using the + composed context. +7. Build and validate the IR. + +For root `R` including `A` then `B`, where `A` includes `C`, the composition +order is: + +```text +C, A, B, R +``` + +This order does not imply that later declarations replace earlier ones. +Duplicates remain errors. + +### 6.1 Repeated includes + +Including the same canonical fragment more than once is an error, even when it +would enter through different relative spellings or symlinks. The +composition-wide `visited` map catches this after the first load, even when the +first identity is no longer on the recursion stack. The duplicate-identity +diagnostic shows both the chain that first loaded the fragment and the +attempted chain that reached it again. + +A future bundle RFC may permit multiple instantiated copies through distinct +namespaces and parameters. Plain includes model source decomposition, not +instantiation. + +### 6.2 Cycle diagnostics + +A cycle reports the complete include chain in lexical include order: + +```text +Netsukefile -> build/common.yaml -> build/rust.yaml -> build/common.yaml +``` + +The error identifies the closing include site and the first active site for the +same canonical identity. + +## 7. Deterministic merge semantics + +Composition operates on typed manifest sections rather than generic YAML map +merging. + +### 7.1 Named declarations + +Rules, targets, and actions are keyed by their final qualified names. Two +composition units may not contribute the same final name. + +The error must include: + +- the duplicate qualified name; +- both source files and source spans; +- the include chains that made each declaration visible; and +- a remediation such as adding `as`, renaming one declaration, or extracting a + parameterized bundle under RFC 0003. + +### 7.2 Variables + +Within one namespace, duplicate variable keys are errors. Nested mappings are +not deep-merged implicitly. A variable mapping is one declared value with one +source of truth. + +This refusal is intentional. Generic deep merge requires policy for sequences, +nulls, scalar-versus-mapping conflicts, and ordering. Silent last-writer-wins +would make include reordering behaviourally significant in ways that are hard +to review. + +Manifest authors may construct derived mappings explicitly with Jinja filters +such as a future deterministic `combine` helper. + +### 7.3 Root-only settings + +Settings that control the complete compilation, including manifest-format +selection and workspace-wide defaults, may appear only in the root manifest +unless their schema explicitly declares fragment scope. + +A fragment containing a root-only setting is rejected at that fragment's source +span. It is not ignored and does not compete by composition order. + +### 7.4 Sequence sections + +Any schema section whose order is semantically meaningful concatenates in the +composition order defined in section 6. The schema must identify such sections +explicitly; implementations must not apply generic YAML sequence concatenation +to unknown fields. + +## 8. Jinja and control-key semantics + +Includes resolve before ordinary manifest-time Jinja evaluation. Include paths +therefore cannot depend on variables, the environment, the filesystem beyond +the literal path, the clock, or network helpers. + +After composition: + +- root and fragment variables form one typed context subject to namespace and + duplicate rules; +- `foreach` and `when` evaluate at their existing permitted scopes; +- metadata queries use the established restricted Jinja environment; and +- recipe-only helpers remain unavailable during `help targets`. + +A fragment cannot use Jinja to create another `includes` key after parsing. +Structural composition remains visible in YAML source. + +## 9. Path and capability semantics + +Each include is opened relative to a directory capability for the including +file. Lexical `..` segments are normalized before access, and symlinks are +resolved before the workspace-boundary check. + +The initial implementation rejects: + +- absolute include paths; +- paths that escape the effective workspace root; +- paths containing NUL; +- non-file objects; +- unreadable files; and +- non-UTF-8 manifest paths or content where the existing manifest contract + requires UTF-8. + +Diagnostics should distinguish missing, unreadable, outside-workspace, +symlink-escape, duplicate-load, and cycle failures. + +## 10. Provenance, hashing, and generated state + +Every declaration and rendered field retains: + +- physical source file; +- source span; +- include chain; +- applied namespace; and +- composed declaration identity. + +The compiled-manifest fingerprint includes, in composition order: + +- the relative canonical identity of every composition unit; +- the complete bytes of every unit; +- namespace selections; and +- the manifest schema and compiler versions. + +Touching metadata without changing bytes does not alter the fingerprint. +Changing included bytes does. + +Generated Ninja and sidecar diagnostics must point to original fragment spans, +not to a synthetic merged YAML document. + +## 11. Metadata and agent-facing behaviour + +`netsuke help targets`, graph inspection, and JSON metadata must expose the +composed declarations while retaining their origin. + +Structured output should include bounded fields such as: + +```json +{ + "name": "rust_quality::lint", + "source": "build/netsuke/rust-quality.yaml", + "namespace": "rust_quality" +} +``` + +It must not expose absolute host paths when a workspace-relative path is +sufficient. + +Metadata discovery resolves and parses includes but does not render or execute +recipe fields. A malformed or cyclic include graph is still a metadata error. + +## 12. Compatibility and migration + +Manifests without `includes` retain their current behaviour. + +A monolithic manifest may migrate mechanically: + +1. move one coherent declaration group into a fragment; +2. add a literal include path; +3. add a namespace if names would collide; +4. update external references to qualified names; and +5. compare graph and help snapshots before removing the original declarations. + +Because duplicate declarations are errors, migration cannot accidentally leave +both the copied and extracted version active. + +## 13. Implementation plan + +### Phase 1: composition AST + +- Add literal `includes` syntax and deny unknown fields. +- Retain source-file and source-span provenance for every parsed unit. +- Implement capability-relative path resolution and canonical identity. + +### Phase 2: resolver and cycle validation + +- Implement deterministic depth-first post-order traversal. +- Add active-stack cycle detection and repeated-identity rejection. +- Add namespace qualification and internal-reference rewriting. + +### Phase 3: typed merge + +- Compose schema sections explicitly. +- Reject duplicate variables and named declarations with dual-source + diagnostics. +- Enforce root-only settings. + +### Phase 4: compiler and metadata integration + +- Feed the composed AST into existing Jinja, `foreach`, `when`, IR, help, graph, + and Ninja generation stages. +- Include all fragments and namespaces in graph fingerprints. + +### Phase 5: documentation and migration canaries + +- Add user and developer guidance. +- Split a representative downstream Netsukefile into local fragments and prove + graph equivalence. + +## 14. Test strategy + +The implementation must include: + +- relative-path resolution from nested directories; +- root, nested, and sibling includes; +- deterministic order snapshots; +- duplicate declaration and variable diagnostics; +- namespace qualification and internal-reference tests; +- lexical and symlink workspace escapes; +- direct, indirect, and symlink-mediated cycles; +- repeated includes through different spellings; +- metadata queries that do not evaluate recipes; +- source-span diagnostics from included files; +- graph-fingerprint changes for byte changes but not metadata-only changes; +- property tests generating bounded acyclic and cyclic include graphs; and +- Windows path and case-behaviour tests. + +A Kani harness may check the bounded traversal state machine and the invariant +that every emitted composition unit appears exactly once. + +## 15. Alternatives considered + +### 15.1 YAML merge keys + +YAML anchors and merge keys operate inside one parsed document, have awkward +cross-file semantics, and do not preserve Netsuke declaration identities or +include provenance. Rejected. + +### 15.2 Jinja include/import + +Template-level includes evaluate too late, mix source composition with value +rendering, and could make metadata queries execute build-time helpers. Rejected +for manifest structure. + +### 15.3 Last-writer-wins map merging + +This is compact but makes include order an implicit override language and hides +stale duplicate declarations. Rejected for the initial surface. + +### 15.4 Globbed includes + +Directory iteration and broad glob capability complicate order, review, +provenance, and injection boundaries. Explicit literal paths are adequate for +source decomposition. Rejected. + +## 16. Open questions + +- Should fragments use a distinct top-level `kind: fragment` marker, or is being + reached through `includes` sufficient? +- Should an include namespace also qualify pools and future provider names? +- Should tooling offer a formatter-assisted extraction command? +- Which root settings, if any, should later gain explicit fragment scope? + +These questions may refine the schema but must not weaken literal path +resolution, duplicate rejection, or deterministic composition order. + +## 17. Recommendation + +Adopt literal, repository-relative includes with depth-first post-order +composition, workspace capability confinement, retained provenance, optional +namespaces, and duplicate rejection. + +This is enough to split large Netsukefiles safely. It intentionally leaves +version selection, parameterization, and network trust to later RFCs rather +than smuggling a package manager into an `include` key. diff --git a/docs/rfcs/0003-versioned-local-bundles.md b/docs/rfcs/0003-versioned-local-bundles.md new file mode 100644 index 000000000..dfc22eb37 --- /dev/null +++ b/docs/rfcs/0003-versioned-local-bundles.md @@ -0,0 +1,703 @@ +# RFC 0003: Versioned local manifest bundles + +## Preamble + +- **RFC number:** 0003 +- **Status:** Proposed +- **Created:** 2026-08-26 +- **Target:** Reusable, parameterized Netsuke manifest composition +- **Depends on:** RFC 0002 repository-relative includes + +## 1. Summary + +This RFC builds a reusable bundle model on top of repository-relative includes. +A bundle is a local directory with explicit identity, semantic version, +parameters, exports, compatibility requirements, content provenance, and one +entry manifest. + +Bundles remain local in this RFC. Netsuke performs no network access and does +not clone repositories. A bundle may be vendored into the current repository, +checked out as part of a larger workspace, or supplied through an explicitly +selected local bundle catalogue. + +The central rule is: + +> A bundle instance is identified by requested source and bundle identity, +> declared version, namespace, and rendered parameters; none of those inputs +> may be inferred from mutable ambient state. + +The design supports two local selection forms: + +- a direct path to one bundle, with an exact version assertion; and +- a local catalogue containing several semantic versions, with deterministic + highest-compatible selection. + +A lock record captures the selected local path, bundle identity, version, +parameter digest, and canonical content digest. This makes local composition +reviewable and prepares the provenance model for the external Git boundary in +RFC 0004. + +## 2. Problem + +RFC 0002 splits a large Netsukefile into local fragments but deliberately +rejects repeated inclusion and duplicate names. That is correct for source-file +decomposition, but it does not solve reusable quality-gate packages such as: + +- a standard Rust format, lint, rustdoc, Whitaker, test, and audit suite; +- a Python format, Ruff, type-check, and pytest suite; +- a release packaging workflow; +- a documentation and spelling bundle; or +- an organization-wide CI contract instantiated with repository-specific + feature lanes and tool versions. + +Copying fragments between repositories loses version identity. Adding implicit +overrides makes updates difficult to review. A reusable bundle needs a public +contract describing what it imports, what it exports, which parameters it +accepts, and which Netsuke versions it supports. + +Semantic version selection also needs a deterministic local rule. Choosing the +first directory returned by the filesystem, silently accepting an incompatible +bundle, or treating a directory name as authoritative would undermine +reproducibility. + +## 3. Goals and non-goals + +### 3.1 Goals + +This RFC aims to: + +- define a versioned, reusable local bundle format; +- permit typed, declared parameters with defaults and validation; +- define explicit exports rather than exposing every internal declaration; +- instantiate the same bundle more than once under distinct namespaces; +- resolve local semantic versions deterministically; +- verify bundle metadata against selected version constraints; +- retain source, parameter, and bundle provenance through diagnostics and JSON; +- record a canonical content digest and lock information; +- remain fully offline; and +- provide the local semantic foundation for RFC 0004. + +### 3.2 Non-goals + +This RFC does not: + +- fetch or clone a Git repository; +- resolve remote tags, branches, or commits; +- trust a bundle merely because its directory name resembles a version; +- permit undeclared parameters; +- permit a bundle to access declarations outside its import contract; +- silently expose private rules, targets, actions, variables, or providers; +- define cryptographic publisher identity or signature verification; or +- allow floating external dependencies. + +## 4. Bundle layout + +A bundle directory contains one required metadata file and one entry manifest: + +```text +rust-quality/ +├── NetsukeBundle.yaml +├── Netsukefile.bundle.yaml +├── fragments/ +│ ├── lint.yaml +│ └── test.yaml +├── tools/ +│ └── run-quality-check +└── README.md +``` + +`NetsukeBundle.yaml` is not a build manifest. It is a small, strictly parsed +bundle descriptor. `Netsukefile.bundle.yaml` is the entry fragment composed +using RFC 0002 semantics. + +The descriptor has this shape: + +```yaml +bundle: + name: df12.rust-quality + version: 1.4.2 + manifest: Netsukefile.bundle.yaml + runtime_resources: + - path: tools/run-quality-check + executable: true + requires: + netsuke: ">=0.2.0, <0.3.0" + manifest: ">=1.1.0, <2.0.0" + +parameters: + cargo: + type: string + default: cargo + expose: non-secret + features: + type: sequence + default: [] + deny_warnings: + type: bool + default: true + +exports: + actions: + - check-fmt + - lint + - test + - all + variables: + - cargo + - features +``` + +Unknown descriptor keys are errors. The descriptor is parsed without Jinja. + +`runtime_resources` is required, including when its value is an explicit empty +list. It names every regular file that the bundle makes available to a spawned +process or reads as runtime data but that is not reachable from the descriptor +or entry-manifest include graph. Each resource is a mapping with a required +`path` and `executable` boolean. The path is a literal, bundle-relative, +normalized path to one regular file. It may not contain Jinja, glob syntax, +parent traversal, or a symlink escape. A bundle must declare every executable +script, helper binary, configuration file, and data file it uses at runtime. +The declared `executable` value is the resource's platform-independent mode: +the canonical digest writes exactly one mode byte, `0x01` for `true` and `0x00` +for `false`; it never derives this value from host permission bits. The +resolver opens each declared resource through the selected bundle directory +capability and includes its path, mode byte, and exact bytes in the canonical +content digest. Duplicate resource paths are errors. The manifest compiler must +reject a bundle-relative runtime file reference that is neither graph-reachable +nor declared in `runtime_resources`. + +## 5. Import syntax + +### 5.1 Direct bundle path + +A direct import names the expected bundle identity, one bundle directory, and +asserts an exact version: + +```yaml +bundles: + - name: df12.rust-quality + source: + path: build/bundles/rust-quality + version: "=1.4.2" + as: rust + with: + cargo: cargo + features: + - serde + - tracing +``` + +The selected descriptor name and version must satisfy the import's `name` and +version requirements. A direct path does not search sibling directories. + +### 5.2 Local catalogue resolution + +A catalogue import names a directory whose immediate children are candidate +bundle directories: + +```yaml +bundles: + - name: df12.rust-quality + source: + catalogue: build/bundle-catalogue/rust-quality + version: "^1.4" + as: rust +``` + +A possible catalogue is: + +```text +build/bundle-catalogue/rust-quality/ +├── 1.3.9/ +│ └── NetsukeBundle.yaml +├── 1.4.1/ +│ └── NetsukeBundle.yaml +├── 1.4.2/ +│ └── NetsukeBundle.yaml +└── 2.0.0/ + └── NetsukeBundle.yaml +``` + +Netsuke examines immediate children in sorted byte order and classifies each +descriptor before version selection. A candidate is valid only when its +`bundle.name` and `bundle.version` fields are present and strictly parse as a +bundle identifier and Semantic Version, respectively. A missing or malformed +field is a malformed candidate: resolution fails with the candidate's +repository-relative path and parse diagnostic, regardless of the directory +name. Directory names are hints only and cannot repair or exclude malformed +metadata. A valid candidate declaring another name is diagnosed as an +out-of-scope candidate and does not participate in selection. A valid candidate +with the requested name but an incompatible version is simply not selected. + +Selection chooses the highest semantic version satisfying the requirement. +Pre-release versions participate only when the requirement itself admits a +pre-release. Build metadata does not affect precedence. + +### 5.3 Semantic Version requirements + +Requirements are comma-separated comparator sets. Whitespace around commas and +comparators is insignificant. The grammar is: + +```plaintext +requirement = comparator ("," comparator)* +comparator = operator? version | "^" version +operator = "=" | ">" | ">=" | "<" | "<=" +version = complete | abbreviated +complete = major "." minor "." patch ["-" prerelease] ["+" build] +abbreviated = major ["." minor] +``` + +`major`, `minor`, and `patch` are non-negative decimal integers without leading +zeroes, except for zero itself. Pre-release and build identifiers use the +Semantic Versioning 2.0.0 rules. Only a complete three-component version may +include pre-release or build metadata; abbreviated versions are valid only when +neither metadata component is present. A bare three-component version and its +`=` form mean an exact core-version requirement. Because build metadata is +ignored for requirement matching and precedence, `=1.4.2` matches a candidate +declaring `1.4.2+local`. A requirement containing a pre-release identifier +matches only candidates with the same core tuple and exactly the same ordered +pre-release identifiers; build metadata remains ignored for matching and +precedence. A bare `1.4` means `>=1.4.0, <1.5.0`, and a bare `1` means +`>=1.0.0, <2.0.0`. When an operator is present, omitted components are zero, so +`>=1.4` means `>=1.4.0` and `<2` means `<2.0.0`. + +The caret operator admits compatible changes: `^1.4` means `>=1.4.0, <2.0.0`; +`^0.4` means `>=0.4.0, <0.5.0`; and `^0.0.3` means `>=0.0.3, <0.0.4`. A +candidate with a pre-release identifier is excluded unless at least one +comparator in the set contains a pre-release identifier with the same major, +minor, and patch tuple. Thus, `^1.4` excludes `1.4.0-rc.1`, while +`>=1.4.0-rc.1, <1.5.0` admits it. Build metadata is ignored for precedence and +requirement matching. + +The resolver applies every comparator in a set, then selects the highest +matching version after applying the pre-release rule. These selection vectors +are normative: + +| Requirement | Candidates | Result | +| ---------------------- | ----------------------------------- | --------------------- | +| `^1.4` | `1.3.9`, `1.4.1`, `1.9.0`, `2.0.0` | select `1.9.0` | +| `^1.4` | `1.3.9`, `2.0.0` | no compatible version | +| `>=1.4, <2.0` | `1.4.0`, `1.9.9`, `2.0.0` | select `1.9.9` | +| bare `1.4` | `1.4.0`, `1.4.9`, `1.5.0` | select `1.4.9` | +| `^1.4` | `1.4.0-rc.1`, `1.4.1-beta`, `1.4.1` | select `1.4.1` | +| `>=1.4.0-rc.1, <1.5.0` | `1.4.0-rc.2`, `1.4.0-beta` | select `1.4.0-rc.2` | +| `=1.4.2` | `1.4.2+local` | select sole candidate | +| `=1.4.2` | `1.4.2+one`, `1.4.2+two` | ambiguity error | +| `=1.4.2-rc.1` | `1.4.2-rc.1` | select sole candidate | +| `=1.4.2-rc.1` | `1.4.2-rc.2` | no compatible version | +| `=1.4.2-rc.1` | `1.4.2-rc.1+local` | select sole candidate | + +Table 2: Semantic Version requirement selection vectors. + +Malformed descriptors fail during candidate classification, before any vector +is selected. Candidates with valid metadata but another name or an incompatible +version are excluded without changing the highest-compatible selection among +valid candidates. After filtering by declared bundle name and requirement, if +more than one candidate has the highest-precedence matching version, resolution +fails with a deterministic ambiguity diagnostic naming every tied candidate's +repository-relative path and declared version in sorted byte order. This +applies regardless of canonical content digest; a digest cannot break the tie. + +### 5.4 Import fields + +| Field | Type | Default | Meaning | +| ------------------ | ---------------------------- | ------------------- | ----------------------------------- | +| `name` | bundle identifier | required | Expected declared bundle identity. | +| `source.path` | local directory | one source required | Direct bundle directory. | +| `source.catalogue` | local directory | one source required | Directory of candidate bundles. | +| `version` | semantic version requirement | required | Accepted bundle version range. | +| `as` | identifier | required | Namespace for this bundle instance. | +| `with` | parameter mapping | empty | Explicit parameter values. | +| `lock` | `required` or `update` | `required` in CI | Lock disposition. | + +Table 1: Local bundle import fields. + +A bundle namespace must be unique in the root composition. + +## 6. Parameter model + +Parameters are declared in the bundle descriptor and supplied through `with`. +The initial type vocabulary is: + +- `string`; +- `bool`; +- `integer`; +- `path`; +- `sequence`; +- `sequence`; and +- `mapping`. + +Every parameter may define a default. A parameter without a default is +required. Unknown supplied parameters are errors. Values are validated before +the entry manifest's Jinja expressions evaluate. + +Parameter values are redacted in human diagnostics, JSON metadata, and debug +metadata by default. A declaration may opt into exposure only with the exact +annotation `expose: non-secret`; no other annotation exposes a value. This +annotation is an assertion by the bundle author that the value is safe to +display, not a request to infer safety from the parameter name or type. The +future explicit `secret` parameter type remains always redacted and cannot be +combined with `expose: non-secret`. + +Parameter expressions in the importing manifest may use the ordinary pure +manifest context, but they must render to the declared type. They may not +invoke network, subprocess, clock, or unrestricted filesystem helpers. + +Inside a bundle, parameters appear under a dedicated immutable object: + +```jinja +{{ bundle.params.cargo }} +{{ bundle.params.features }} +``` + +A bundle may not mutate a parameter or observe the importer's unrelated +variables. + +The normalized parameter value contributes to bundle instance identity and the +compiled graph fingerprint. + +## 7. Export and namespace semantics + +A bundle is private by default. Only descriptor-listed declarations become +visible to the importer. + +Internal rules, variables, targets, and actions remain available within the +bundle's own composition but cannot be referenced from the root or another +bundle. + +An imported action `lint` from namespace `rust` is exposed as: + +```text +rust::lint +``` + +Internal references are rewritten before global resolution. Exports may refer +to private bundle declarations, but consumers cannot bypass the exported entry +point to reach them. + +Export lists are validated against the composed bundle. Missing exports are +errors. A declaration may not be exported under two names in the initial +surface. + +## 8. Bundle composition + +The entry manifest and its local fragments compose under RFC 0002 with these +additional boundaries: + +- include paths remain relative to the bundle directory; +- symlinks may not escape the bundle root; +- the bundle cannot include files from the importing repository outside its own + directory; +- root-only manifest settings remain forbidden; +- bundle parameters enter before bundle-local Jinja evaluation; and +- only declared exports leave the bundle namespace. + +A bundle may import another local bundle. Nested imports resolve before the +parent bundle, and their namespaces are private to the parent unless explicitly +re-exported by a later RFC. + +The import graph must remain acyclic by canonical bundle instance identity. + +## 9. Version and compatibility semantics + +Bundle versions follow Semantic Versioning 2.0.0. + +The descriptor's `requires.netsuke` and `requires.manifest` requirements are +checked before parsing the entry manifest beyond the minimum needed for safe +diagnostics. + +A bundle import fails when: + +- no candidate satisfies the requested version; +- no candidate has the requested declared bundle name; +- the selected bundle rejects the running Netsuke version; +- the selected bundle rejects the active manifest-format version; +- candidate identity or version is ambiguous; +- an exact direct-path assertion does not match; or +- the lock record selects different content. + +Netsuke must not silently choose an older candidate because a newer compatible +candidate contains malformed metadata. Candidate classification fails before +selection when any descriptor has a missing or malformed `bundle.name` or +`bundle.version`; the failure includes the candidate's repository-relative path +and parse diagnostic. This fail-closed rule applies even when the directory +name would look outside the requested range, because directory names are not +authoritative. Only candidates with strictly valid metadata can be excluded as +out of scope or incompatible and then omitted from highest-compatible selection. + +## 10. Tagged Git relationship + +This RFC remains local-only, but it reserves provenance fields needed for +tagged Git resolution under RFC 0004: + +- requested source kind; +- requested version requirement; +- bundle name and declared version; +- source repository identity when known; +- requested Git tag when supplied by a later resolver; +- tag object identifier for annotated tags; +- peeled commit identifier; +- bundle subdirectory; +- canonical content digest; and +- lock-record version. + +A locally checked-out bundle may carry advisory Git provenance generated by a +vendor/update tool, but local resolution does not invoke Git or trust the +working tree's current tag. The external resolver in RFC 0004 owns exact tag +lookup and verification. + +This separation avoids a surprising rule where the same local files resolve +differently merely because `.git` is present. + +## 11. Canonical content digest + +Netsuke computes a digest over a canonical bundle tree containing: + +- every regular file reachable from the descriptor and entry-manifest include + graph; +- every regular file named by `runtime_resources`; +- normalized paths relative to the selected bundle root, encoded with `/` as + the separator; +- file type metadata and, for each declared runtime resource, exactly one + platform-independent mode byte (`0x01` for executable and `0x00` for + non-executable); and +- exact file bytes. + +The canonical stream sorts those bundle-root-relative paths bytewise and +encodes each record as its normalized path, file-type marker, declared mode +byte when present, and length-prefixed exact bytes. Resource mode bytes come +only from the descriptor's `executable` boolean, never from host filesystem +permission bits. The bundle root is not part of a record, so relocating an +otherwise identical bundle to another workspace or Git subdirectory preserves +its digest. + +The canonicalization excludes: + +- `.git` data; +- filesystem timestamps, owners, and inode numbers; +- files neither reachable from the bundle composition nor named by + `runtime_resources`; and +- generated Netsuke state. + +The initial algorithm is SHA-256 with an algorithm-qualified representation: + +```text +sha256:4e9f... +``` + +The canonical tree format is versioned so a future algorithm or normalization +change cannot silently reinterpret an old digest. + +## 12. Lock records + +A repository using bundles stores deterministic selections in a versioned +Netsuke lock file. One conceptual record is: + +```yaml +bundles: + rust: + name: df12.rust-quality + version: 1.4.2 + source: + kind: local-catalogue + path: build/bundle-catalogue/rust-quality/1.4.2 + parameters_sha256: sha256:8b25... + content_sha256: sha256:4e9f... +``` + +Normal CI and release builds require the lock record to match. An explicit +update command may select a newer compatible version and atomically rewrite the +record. + +The lock file never contains absolute host paths. Paths are relative to the +workspace root or an explicitly named local catalogue root. + +Changing bundle files without changing the declared version is visible as a +content-digest mismatch. Local development may offer a deliberate unlocked +mode, but it must be explicit in human and JSON output and should not be the CI +default. + +## 13. Provenance and diagnostics + +Every declaration originating from a bundle retains: + +- import namespace; +- bundle name and version; +- direct path or catalogue selection; +- descriptor and source file spans; +- parameter source and normalized parameter digest; +- canonical content digest; and +- nested bundle chain. + +Human diagnostics should remain concise. JSON metadata may expose the complete +bounded provenance object without leaking absolute paths or parameter values. +Parameter values remain redacted unless their declaration carries the explicit +`expose: non-secret` annotation. Secret-shaped parameters must be passed +through a future explicit `secret` type, which remains redacted regardless of +annotations. + +## 14. Security and capability boundaries + +Bundle paths and catalogue paths are literal, repository-relative paths opened +through directory capabilities. Symlinks may not escape their selected bundle +or catalogue boundary. + +Bundle metadata and manifests cannot: + +- request network access during composition; +- discover arbitrary sibling files; +- invoke Git; +- read undeclared importer variables; +- enumerate the process environment; or +- export undeclared private implementation details. + +A bundle's recipes retain the same execution capabilities as ordinary manifest +recipes. Bundle provenance does not make an unsafe command safe. + +## 15. CLI and metadata surface + +The following conceptual commands should exist, subject to the CLI vocabulary +decision: + +```console +netsuke bundle list +netsuke bundle inspect rust +netsuke bundle update rust +netsuke bundle verify +``` + +Equivalent placement beneath existing commands is acceptable if it avoids a new +top-level noun. + +Human and JSON output should show: + +- namespace; +- bundle name and selected version; +- source kind and relative path; +- content digest; +- lock status; +- exported declaration names; and +- compatibility requirements. + +Parameter values are redacted by default and may be displayed only when the +schema carries `expose: non-secret`; secret-typed values are always redacted. + +## 16. Compatibility and migration + +RFC 0002 includes remain valid and do not acquire version semantics. + +A local fragment can migrate into a bundle by: + +1. adding a descriptor; +2. naming and versioning the bundle; +3. declaring parameters instead of reading importer variables; +4. listing exports; +5. importing it under a namespace; and +6. generating a lock record. + +A bundle descriptor or import syntax requires an additive manifest-format minor +version. Older Netsuke versions must reject it cleanly. + +## 17. Implementation plan + +### Phase 1: descriptor and types + +- Implement strict bundle descriptor parsing. +- Add semantic version and requirement validation. +- Add typed parameter schemas and normalization. + +### Phase 2: local direct imports + +- Resolve direct paths through capabilities. +- Compose entry manifests under RFC 0002. +- Implement private-by-default exports and namespace rewriting. + +### Phase 3: catalogue resolution + +- Enumerate immediate candidate directories in sorted order. +- Parse descriptors and select the highest compatible stable version. +- Reject duplicate identity/version candidates. + +### Phase 4: digest and lock + +- Define the versioned canonical bundle-tree format. +- Compute content and parameter digests. +- Add atomic lock verification and update flows. + +### Phase 5: metadata and downstream canary + +- Expose bounded provenance in human and JSON output. +- Package one repeated estate quality-gate set as a versioned local bundle and + consume it from at least two downstream repositories. + +## 18. Test strategy + +The implementation must include: + +- direct exact-version success and mismatch tests; +- catalogue highest-compatible selection; +- comparator-set, omitted-component, caret, and pre-release selection vectors; +- malformed candidate and duplicate identity/version failures; +- catalogue candidates with distinct declared names, proving selection uses the + import's required name before comparing compatible versions; +- typed parameter defaults, missing values, type failures, and unknown keys; +- private declaration isolation and explicit export tests; +- multiple instances under distinct namespaces; +- nested bundle cycles; +- symlink and lexical boundary escapes; +- content-digest stability across timestamp changes; +- digest changes for reachable and declared runtime-resource byte, mode, and + path changes; +- resource mode digest changes use descriptor `executable: true` versus + `executable: false`, independent of host permission bits; +- missing, escaping, non-regular, and undeclared runtime-resource failures; +- lock verification and atomic update tests; +- JSON provenance snapshots proving default redaction and explicit + `expose: non-secret` opt-in; and +- property tests over bounded version catalogues and parameter maps. + +Cross-platform tests must cover path spelling and Windows case behaviour +without making version selection depend on host directory enumeration. + +## 19. Alternatives considered + +### 19.1 Treat any included directory as a bundle + +Without metadata, parameters, exports, and compatibility requirements, a +directory is merely a larger fragment. Rejected. + +### 19.2 Use directory names as versions + +Directory names are convenient for humans but not authoritative and can drift +from the bundle's actual contract. Rejected; descriptor versions decide. + +### 19.3 Last-writer-wins customization + +Allowing an importer to redefine bundle variables hides its supported parameter +surface and makes updates fragile. Rejected in favour of declared parameters. + +### 19.4 Resolve local Git tags directly + +A working tree can be detached, dirty, shallow, missing tags, or embedded +without `.git`. Making local composition depend on repository metadata would +produce different results for identical files. Rejected here; RFC 0004 owns +explicit tagged Git resolution. + +## 20. Open questions + +- Should local catalogue roots be declared globally or only per import? +- Should bundle parameters support enums and constrained strings in the initial + implementation? +- Should local lock updates require an explicit `--update` flag or a dedicated + command? +- What stable vocabulary should describe an unlocked development import? + +## 21. Recommendation + +Adopt versioned local bundles as a strict layer above repository-relative +includes: explicit identity, SemVer, typed parameters, private-by-default +exports, deterministic local resolution, canonical digests, and lock records. + +Do not couple this local model to ambient Git state or networking. That clean +boundary lets RFC 0004 add exact tagged Git resolution and external provenance +without changing what a bundle means. diff --git a/docs/rfcs/0004-digest-pinned-external-bundles.md b/docs/rfcs/0004-digest-pinned-external-bundles.md new file mode 100644 index 000000000..eea785880 --- /dev/null +++ b/docs/rfcs/0004-digest-pinned-external-bundles.md @@ -0,0 +1,643 @@ +# RFC 0004: Digest-pinned external bundles and Git provenance + +## Preamble + +- **RFC number:** 0004 +- **Status:** Proposed +- **Created:** 2026-08-26 +- **Target:** Later external bundle acquisition and provenance +- **Depends on:** RFC 0002 and RFC 0003 + +## 1. Summary + +This RFC defines the later network boundary for Netsuke manifest bundles. It +adds external Git bundle sources only after repository-relative includes and +versioned local bundles have stable semantics. + +An external bundle reference must identify an exact Git object and an expected +canonical bundle digest. A human-friendly tag may be used as the requested +reference, but Netsuke resolves that exact tag to immutable object identifiers, +peels annotated tags deterministically, verifies bundle metadata and content, +and records the result in a lock file. + +Branches, default branches, `HEAD`, unqualified symbolic revisions, and mutable +version-only references are not reproducible sources and are rejected by the +initial external resolver. + +The central rule is: + +> A Git tag is a discovery handle, not the trust anchor. The lock record and +> canonical content digest bind the selected bytes. + +The initial supported external source is a bounded Git repository and optional +bundle subdirectory. Acquisition is explicit through `netsuke bundle fetch`, +cacheable, offline-verifiable, and observable through redacted provenance +metadata. + +## 2. Sequencing + +This RFC is intentionally third in the composition sequence: + +1. RFC 0002 defines deterministic local source composition. +2. RFC 0003 defines bundle identity, versions, parameters, exports, canonical + content digests, and locks without networking. +3. This RFC adds network acquisition, exact Git tag resolution, cache policy, + and external provenance. + +Implementation must not begin by adding `git clone` to the include resolver. +The local semantic model must exist first so remote acquisition supplies bytes +to an already-defined bundle verifier rather than defining bundle meaning by +accident. + +Acceptance of this RFC may occur during v0.2.0 design work. External +acquisition itself may ship later than local includes and local bundles. + +## 3. Problem + +Reusable quality-gate and workflow bundles become much more valuable when +repositories can consume a reviewed upstream release rather than copying or +vendoring every update manually. + +A naïve Git include creates several hazards: + +- a branch or tag may move; +- annotated and lightweight tags resolve differently; +- short revision names may be ambiguous; +- a repository may contain multiple bundles or unrelated files; +- submodules can trigger additional network access; +- credentials may leak through URLs or diagnostics; +- a shallow fetch may resolve a different object set from a full clone; +- cached content may become detached from its origin; +- Git object identity alone does not specify path normalization or the selected + bundle subdirectory; and +- network availability can make graph generation nondeterministic. + +A version requirement such as `^1.4` is also insufficient by itself. Semantic +version metadata is authored by the bundle publisher and does not prove which +bytes were selected. + +Netsuke needs a resolver that allows convenient tagged releases while binding +execution to immutable, digest-verified content. + +## 4. Goals and non-goals + +### 4.1 Goals + +This RFC aims to: + +- resolve exact Git tags, including annotated and lightweight tags; +- support exact commit references as a lower-level alternative; +- peel annotated tags and record both tag-object and commit identities; +- require a canonical bundle digest for external content; +- verify bundle identity, declared semantic version, and compatibility metadata; +- record complete bounded provenance in a versioned lock file; +- provide content-addressed caching and deterministic offline reuse; +- make network access explicit and policy-controlled; +- reject mutable or ambiguous revisions; +- avoid shelling out to Git pipelines; and +- keep credentials and absolute cache paths out of manifests and diagnostics. + +### 4.2 Non-goals + +The initial external resolver does not: + +- resolve branches, remote default branches, `HEAD`, or date-based revisions; +- execute arbitrary Git credential helpers without explicit policy; +- initialize submodules; +- use Git LFS smudge filters; +- run repository hooks, filters, or checkout scripts; +- verify publisher identity solely from a Git tag; +- define a public multi-tenant bundle registry; +- provide dependency solving across unrelated bundle graphs; +- allow metadata queries, including `netsuke help targets`, to fetch external + bundles, publish cache entries, or update lock files; +- mutate the lock file during an ordinary build; or +- treat transport encryption as content authentication. + +Signed-tag and transparency-log policy may be layered on later. The mandatory +initial trust anchor remains the canonical digest in reviewed repository state. + +## 5. Manifest syntax + +An external bundle import extends RFC 0003's `source` union: + +```yaml +bundles: + - source: + git: + url: https://github.com/df12/netsuke-bundles.git + tag: rust-quality/v1.4.2 + subdir: bundles/rust-quality + digest: sha256:4e9f2d... + name: df12.rust-quality + version: "=1.4.2" + as: rust + with: + features: + - serde + - tracing +``` + +An exact commit may replace the tag: + +```yaml +source: + git: + url: https://github.com/df12/netsuke-bundles.git + commit: 8c5e4e6d76d75de67f66f09cb2c63d69d2e14f84 + subdir: bundles/rust-quality + digest: sha256:4e9f2d... +``` + +Exactly one of `tag` or `commit` is required. + +### 5.1 Git source fields + +| Field | Type | Default | Meaning | +| --------------- | -------------------------- | --------------------- | -------------------------------------------------------- | +| `url` | canonical Git URL | required | Repository transport identity. | +| `tag` | exact tag name | one revision required | Human-friendly exact tag to resolve. | +| `commit` | full object ID | one revision required | Exact commit to use directly. | +| `subdir` | relative directory | repository root | Bundle directory within the selected tree. | +| `digest` | algorithm-qualified digest | required | RFC 0003 canonical bundle content digest. | +| `tag_object` | full object ID | absent | Optional reviewed assertion for an annotated tag object. | +| `peeled_commit` | full object ID | absent | Optional reviewed assertion for the resolved commit. | +| `allow_shallow` | Boolean | `true` | Permit a bounded exact-ref fetch when supported. | + +Table 1: External Git source fields. + +`tag_object` and `peeled_commit` are normally written to the lock file rather +than hand-authored. Supplying them in the manifest is an additional assertion, +not a replacement for the digest. + +Unknown keys are errors. Jinja is forbidden in `url`, `tag`, `commit`, +`subdir`, and `digest`. + +## 6. Exact tag resolution + +### 6.1 Reference normalization + +A short tag value such as: + +```text +rust-quality/v1.4.2 +``` + +is normalized only to: + +```text +refs/tags/rust-quality/v1.4.2 +``` + +It is never searched across branches, remote-tracking refs, notes, pull-request +refs, or other namespaces. + +A value already beginning with `refs/tags/` is accepted after validation. Other +`refs/` namespaces are rejected in the `tag` field. + +Tag names must satisfy Git's ref-format rules, contain no NUL, and be bounded +in length. Revision expressions such as `^{}`, `~1`, `^1`, `:path`, or reflog +selectors are not tag names and are rejected. + +### 6.2 Fetch contract + +The `netsuke bundle fetch` command requests the exact normalized tag ref from +the configured repository. It must not fetch every branch or rely on the +remote's default ref. No metadata query or ordinary build may invoke this fetch +contract. + +The resolver may use an embedded Git implementation or a tightly controlled Git +subprocess adapter. In either case it must: + +- avoid shell command construction; +- pass arguments as an argv vector; +- disable hooks and checkout filters; +- avoid writing into the user's working tree; +- bound object, pack, and total transfer sizes; +- enforce configured transport and host policy; +- redact credentials from logs; and +- report the exact requested ref and repository identity. + +### 6.3 Annotated tags + +When the fetched tag ref points to a tag object, Netsuke records: + +- the tag object ID; +- the tag's declared target type and target object ID; +- the complete peel chain, bounded to reject cycles or unreasonable depth; and +- the final commit ID. + +The initial implementation accepts tag chains only when every hop is a tag +object and the final object is a commit. A tag ultimately naming a tree, blob, +or non-commit object is rejected. + +### 6.4 Lightweight tags + +When the tag ref points directly to a commit, the tag is lightweight. The lock +record stores no tag-object ID and records the direct commit as the peeled +commit. + +Human and JSON output distinguish annotated and lightweight tags. Netsuke does +not imply that either form is cryptographically signed. + +### 6.5 Retagging + +If a requested tag resolves to a different object than the lock record, normal +build and verification fail even when the resulting canonical bundle digest is +unchanged. + +An explicit update operation may review and record the new tag and commit +identities. The update still requires the manifest digest to match or an +explicit digest change in reviewed source. + +This detects retagging as a provenance event rather than silently accepting it. + +## 7. Exact commit resolution + +The `commit` form requires the full object ID appropriate to the repository's +hash algorithm. Abbreviated IDs are rejected. + +The selected object must be a commit. Netsuke fetches that exact object through +a bounded protocol where the server supports it. If the server refuses +unadvertised object fetches, the diagnostic must explain the limitation rather +than broadening the fetch to mutable branches automatically. + +The lock record notes that no tag was requested. Bundle semantic version checks +still apply to the selected subdirectory's descriptor. + +## 8. Bundle verification + +After resolving a commit, Netsuke reads the selected tree without performing a +normal checkout where practical. + +It then: + +1. validates `subdir` as a repository-relative, non-escaping path; +2. traverses the selected Git tree one component at a time, accepting only + directory entries and rejecting any symlink component before locating + `NetsukeBundle.yaml`; the descriptor entry itself must also be a regular + file, not a symlink; +3. locates `NetsukeBundle.yaml` in that directory; +4. parses and validates the RFC 0003 descriptor; +5. verifies bundle name and semantic version against the import; +6. verifies Netsuke and manifest-format compatibility; +7. computes the RFC 0003 canonical content digest over graph-reachable bundle + content and every regular file declared by `runtime_resources`; and +8. compares it with the mandatory manifest digest and lock record. + +A matching Git commit with a mismatched canonical digest fails. A matching +digest under a different unexpected tag or commit also fails normal lock +verification. + +The digest binds the selected bundle content rather than the complete upstream +repository. Unrelated documentation or bundles elsewhere in the repository do +not alter it unless the selected bundle declares them as graph-reachable +content or `runtime_resources`. + +## 9. Lock-file provenance + +One conceptual external lock record is: + +```yaml +bundles: + rust: + name: df12.rust-quality + version: 1.4.2 + source: + kind: git + url: https://github.com/df12/netsuke-bundles.git + requested_tag: refs/tags/rust-quality/v1.4.2 + tag_kind: annotated + tag_object: 91ca... + peeled_commit: 8c5e4e6d76d75de67f66f09cb2c63d69d2e14f84 + subdir: bundles/rust-quality + parameters_sha256: sha256:8b25... + content_sha256: sha256:4e9f2d... +``` + +The lock record also stores: + +- repository object-format algorithm; +- canonical bundle-tree format version; +- resolver implementation/version; +- acquisition timestamp only as informational metadata excluded from + reproducibility comparisons; and +- any selected verification policy result. + +Normal builds and metadata queries never update this record. +`netsuke bundle fetch` preserves existing lock files unless the caller supplies +the explicit `--update-lock` flag. With that flag, the command may update the +record only after verification, and its human and JSON output shows old and new +tag, commit, version, and digest values. + +## 10. Network and offline policy + +Metadata queries and ordinary builds are cache-only in every mode. They may +read a verified content-addressed cache entry, but they must never initiate +network access, publish a cache entry, or update a lock file. This includes +`netsuke help targets`. + +`netsuke bundle fetch` is the sole command permitted to initiate external +network acquisition or publish a verified cache entry. Its conceptual modes are: + +- `offline`: use only verified content-addressed cache entries and fail without + contacting the network; +- `locked`: acquire only the identities and digest required by the existing + lock record, without changing that record; +- `update`: resolve requested tags anew, with lock changes permitted only when + `--update-lock` is also supplied; and +- `refresh`: refetch locked identities without changing selection or the lock + record. + +Without `--update-lock`, `netsuke bundle fetch` preserves existing lock files +in every mode. A missing verified cache entry produces a typed cache-miss +diagnostic naming the bundle namespace and expected digest; metadata queries +return that diagnostic rather than attempting acquisition. + +## 11. Content-addressed cache + +External bundle content is stored under a content-addressed key derived from: + +- canonical content digest; +- canonical bundle-tree format version; and +- bundle metadata schema version. + +Repository transport data and Git object packs may use a separate bounded cache +keyed by canonical repository identity and object IDs. + +Cache entries are written atomically, owner-protected, and verified before use. +A cache hit never bypasses descriptor, version, digest, or lock validation. + +Concurrent processes coordinate through leases rather than trusting partial +files. Stale temporary entries are cleaned through bounded scans. + +Cache locations are configuration, not manifest semantics, and never appear as +absolute paths in generated metadata. + +## 12. Repository identity and URL handling + +Netsuke canonicalizes repository identity without rewriting across unrelated +hosts or protocols. The initial accepted URL forms should be explicit, for +example: + +- `https://host/owner/repository.git`; +- `ssh://user@host/path/repository.git`; and +- a configured enterprise Git transport. + +SCP-like syntax may be deferred unless a robust parser is available. + +Credentials embedded in URLs are rejected. Authentication enters through a +credential provider or transport configuration at the composition root. +Credentials are never written to the lock file, cache metadata, diagnostics, +telemetry, or generated plans. + +Redirect policy is bounded and records the final canonical repository identity. +Cross-host redirects require explicit policy. + +## 13. Submodules, LFS, and repository features + +The initial resolver does not initialize submodules and treats Gitlinks within +the selected bundle content as unsupported reachable entries. + +Git LFS pointer files remain ordinary bytes. Netsuke does not run smudge +filters. A bundle requiring LFS materialization is rejected unless a later RFC +defines an explicit, digest-verifiable LFS source contract. + +Repository hooks, clean/smudge filters, sparse-checkout configuration, and +attributes that invoke external processes are disabled or ignored during +acquisition. + +The canonical bundle digest is computed from selected Git tree content under +Netsuke's own normalization, not from a user checkout affected by filters. + +## 14. Trust and verification + +The mandatory trust chain is: + +1. reviewed manifest source names a repository, exact tag or commit, semantic + version requirement, subdirectory, and canonical digest; +2. lock source records exact resolved object identities; +3. acquisition retrieves those objects under bounded transport policy; +4. bundle metadata and compatibility are validated; and +5. canonical selected content must match the reviewed digest. + +The cache-only rule is also a security boundary: metadata queries cannot turn +an informational command into an unreviewed network fetch or cache publication. +Only `netsuke bundle fetch` may cross that boundary, and lock changes require +the explicit `--update-lock` flag. + +An annotated tag signature may provide additional publisher identity. A future +policy may require: + +- a valid signature from an allowed key; +- an allowed signing identity; +- a transparency-log inclusion proof; or +- a repository-host attestation. + +Signature verification is additive. It does not replace digest verification, +because a validly signed tag can still point to content different from what the +consumer reviewed. + +## 15. Provenance and agent-facing output + +Human inspection should show a compact provenance summary: + +```text +rust: df12.rust-quality 1.4.2 + git tag rust-quality/v1.4.2 + commit 8c5e4e6d76d75de67f66f09cb2c63d69d2e14f84 + digest sha256:4e9f2d... + lock verified, cache hit +``` + +JSON output may include: + +- requested and normalized tag; +- annotated/lightweight classification; +- tag-object and peeled-commit IDs; +- repository object format; +- subdirectory; +- bundle identity/version; +- content and parameter digests; +- lock status; +- cache status; +- network mode; and +- verification-policy result. + +Metrics use bounded categories only. Repository URLs, tags, commits, bundle +names, paths, and digests must not become unbounded metric labels. + +## 16. Failure model + +Typed errors should distinguish: + +- invalid or unsupported URL; +- disallowed credentials or transport; +- network disabled; +- exact tag absent; +- ambiguous or invalid tag name; +- unsupported tag target or excessive peel depth; +- exact commit unavailable; +- object-format mismatch; +- transfer or object-size limit exceeded; +- unsupported Gitlink/submodule; +- missing bundle descriptor or invalid subdirectory; +- semantic version mismatch; +- bundle compatibility mismatch; +- canonical digest mismatch; +- lock tag or commit mismatch; +- verified cache miss naming the bundle namespace and expected digest; +- cache corruption; and +- verification-policy failure. + +Diagnostics must redact credentials and avoid dumping remote protocol payloads. + +## 17. Compatibility and migration + +RFC 0002 includes and RFC 0003 local bundles remain entirely local and do not +acquire network behaviour. + +A vendored local bundle may migrate to an external source by: + +1. publishing the unchanged bundle under a reviewed Git tag; +2. computing its canonical RFC 0003 digest; +3. replacing the local source with the Git source; +4. retaining the same namespace, parameters, bundle identity, and semantic + version assertion; +5. generating and reviewing the external lock record; and +6. proving graph equivalence before removing the vendor copy. + +The external source syntax requires an additive manifest-format minor version. +Older Netsuke versions must reject it clearly. + +## 18. Implementation plan + +### Phase 0: prerequisite stability + +- Accept and implement RFC 0002 local includes. +- Accept and implement RFC 0003 bundle descriptors, parameters, exports, + canonical digests, and locks. + +### Phase 1: Git object resolver + +- Parse and validate exact Git source syntax. +- Implement canonical tag normalization and exact ref fetch. +- Support annotated-tag peeling and lightweight tags. +- Support exact full commit references. + +### Phase 2: bounded acquisition + +- Add transport policy, limits, redaction, and owner-protected temporary state. +- Disable hooks, filters, submodules, and ambient checkout behaviour. +- Read selected tree content without a normal checkout where practical. + +### Phase 3: verification and locking + +- Feed selected content through the RFC 0003 verifier. +- Add complete external provenance to lock records. +- Detect retagging and object-identity drift. + +### Phase 4: content-addressed cache and offline mode + +- Add atomic verified caches, leases, and bounded stale cleanup. +- Implement cache-only metadata queries and the explicit `netsuke bundle fetch` + command with offline, locked, update, and refresh modes. + +### Phase 5: policy and canaries + +- Add optional signed-tag policy behind explicit configuration. +- Consume one digest-pinned tagged bundle from a downstream canary and reproduce + its graph in offline mode. + +## 19. Test strategy + +The implementation must include: + +- exact lightweight and annotated tag resolution; +- nested annotated tag peeling and bounded-depth rejection; +- invalid tag names and revision-expression rejection; +- tag-to-non-commit rejection; +- exact commit success and abbreviated-ID rejection; +- retagging detection with matching and differing content digests; +- semantic version and descriptor mismatch tests; +- subdirectory traversal and symlink/Gitlink boundary tests; +- digest verification over selected content only; +- credentials and redirect policy tests; +- transfer, object, pack, and repository-size limits; +- disabled hooks, filters, submodules, and LFS behaviour; +- lock update snapshots showing old/new provenance; +- cache corruption, atomic publication, concurrency, and stale cleanup; +- offline cache-hit and cache-miss behaviour; +- metadata queries that read verified cache entries only and return typed + cache-miss errors without network, cache-publication, or lock side effects; +- `netsuke bundle fetch` acquisition and verified cache publication; +- ordinary fetch lock preservation and explicit `--update-lock` behaviour; +- SHA-1 and SHA-256 Git object-format fixtures where supported; and +- end-to-end tagged Git resolution against a local protocol test server rather + than the public internet. + +Property tests should cover tag normalization, ref rejection, lock comparison, +and bounded provenance serialization. + +## 20. Alternatives considered + +### 20.1 Branch references + +Branches are mutable and make ordinary builds perform change detection. +Rejected for the initial reproducible source model. + +### 20.2 Tag without digest + +A tag can be retargeted and says nothing about Netsuke's selected-subtree +canonicalization. Rejected. + +### 20.3 Commit without digest + +A commit is immutable in ordinary Git semantics, but the digest also binds the +bundle subdirectory and canonical reachable-resource model independently of Git +transport and object format. Exact commits remain supported, but the bundle +digest is still required. + +### 20.4 Download release archives + +Archives can work but have host-specific generation, redirect, content-type, +and path-normalization concerns. Git tagged resolution provides stronger object +provenance for the initial external source. A digest-pinned archive source may +be proposed separately. + +### 20.5 Trust signed tags alone + +Signatures identify a signer under a policy; they do not prove the consumer +reviewed the selected bytes. Rejected as a replacement for digest pinning. + +### 20.6 Use a normal checkout + +A checkout can invoke filters, depend on user configuration, materialize +submodules, and introduce filesystem metadata. Rejected as the normative +verification representation. + +## 21. Open questions + +- Which embedded Git implementation or controlled subprocess boundary best + satisfies SHA-256 repository support and bounded exact-ref fetches? +- Should annotated tags be required by policy for organization-owned bundles? +- Should external bundle caches share infrastructure with `fetch()` resources + or remain isolated by trust class? +- Should a later registry map semantic versions to digest-pinned Git tags, or is + direct repository metadata sufficient? +- How should enterprise-host identity and credential-provider configuration be + represented without entering manifest semantics? + +## 22. Recommendation + +Adopt external Git bundles only as digest-pinned, lock-recorded acquisitions +over RFC 0003 bundle semantics. + +Support exact human-friendly tags, including annotated and lightweight tags, +but resolve them to immutable object identities and treat retagging as a +reviewable provenance change. Reject branches and implicit network access; +reserve network acquisition and cache publication for the explicit +`netsuke bundle fetch` command, with lock updates gated by `--update-lock`. + +This gives Netsuke tagged Git resolution without asking users to pretend that a +tag is immutable, signed, or sufficient on its own. diff --git a/docs/rfcs/0009-structured-command-working-directories.md b/docs/rfcs/0009-structured-command-working-directories.md new file mode 100644 index 000000000..64f3f3cd0 --- /dev/null +++ b/docs/rfcs/0009-structured-command-working-directories.md @@ -0,0 +1,512 @@ +# RFC 0009: Structured-command working directories + +## Preamble + +- **RFC number:** 0009 +- **Amends:** RFC 0001, Structured command blocks and argv templates +- **Status:** Proposed +- **Created:** 2026-08-26 +- **Target:** Structured command block schema and execution IR + +## 1. Summary + +This amendment adds a first-class `cwd` field to RFC 0001 structured command +blocks. + +The field selects the child process working directory without embedding `cd`, +`pushd`, platform shell syntax, or directory state in a legacy command group. +It applies equally to direct and explicit-shell structured commands and may +vary between stages of one structured pipeline. + +Upon acceptance, RFC 0001 must be read as if: + +- per-command working directories were removed from its non-goals; +- the `cwd` field appeared in the structured command schema and execution IR; +- executable resolution, spawning, validation, diagnostics, security, and tests + included the semantics below; and +- the open question asking whether `cwd` should exist were resolved in favour + of this amendment. + +The implementation PR should fold this amendment into the main RFC text before +RFC 0001 moves from Proposed to Accepted. + +## 2. Motivation + +Several downstream Makefiles operate across more than one project root. Common +examples include: + +- running Cargo commands in `rust_extension`; +- running backend checks below `backend/`; +- invoking tooling from a generated fixture directory; +- building one workspace member with a tool whose configuration is relative to + that member; and +- executing a pipeline whose producer and consumer belong to different local + subprojects. + +Without `cwd`, a structured command must either: + +- fall back to `shell: true` and write `cd directory && command`; +- wrap the command in a helper script; or +- require every called tool to expose an equivalent directory option. + +That undermines the shell-free path RFC 0001 is intended to provide. Working +directory is a primitive of every process API and belongs alongside argv, +environment, and standard streams. + +## 3. Goals and non-goals + +### 3.1 Goals + +This amendment aims to: + +- set the child process working directory explicitly; +- preserve direct-mode argv safety; +- use the same field in direct and shell modes; +- support a distinct directory for each pipeline stage; +- keep path resolution deterministic and capability-scoped; +- retain the existing stream-path and graph-path contract unless stated + explicitly; +- expose useful source provenance and diagnostics; and +- preserve all legacy command semantics. + +### 3.2 Non-goals + +This amendment does not: + +- make `cwd` persist to a later structured block; +- alter the current one-shell state sharing inside a contiguous legacy command + group; +- infer target inputs or outputs from the working directory; +- permit arbitrary directory escape outside the effective workspace; +- create a missing directory; +- search upward for a workspace, manifest, or tool configuration; +- change the base used to resolve `stdin`, `stdout`, `stderr`, or `tee` paths; +- define a directory stack; or +- add per-rule, per-target, or per-action enclosing working directories. + +Those enclosing scopes may be proposed later with explicit precedence rules. + +## 4. Schema amendment + +RFC 0001 section 6.2 gains `cwd`: + +```yaml +invoke: cargo clippy --all-targets +shell: false +cwd: rust_extension + +env: + RUSTFLAGS: -D warnings + +stdin: input.txt +stdout: output.txt +stderr: errors.txt +tee: trace.log +pipe: false +``` + +The field table gains: + +| Field | Type | Default | Meaning | +| ----- | ----------- | ----------------------------------- | -------------------------------- | +| `cwd` | string path | Netsuke effective working directory | Child process working directory. | + +Table 1: Structured-command fields. + +Unknown-key rejection remains unchanged. + +`cwd` is valid on every structured command block, whether singular, in a +heterogeneous sequence, or in a structured pipeline. + +## 5. Rendering and path resolution + +`cwd` is a scalar Jinja template rendered at manifest compilation time. It must +produce one non-empty string containing no NUL. + +Sequence, mapping, null, undefined, and callable values are errors. + +A relative `cwd` is resolved against Netsuke's effective working directory +after CLI `-C` processing. It is not resolved relative to: + +- the source file containing the command; +- a bundle directory; +- the previous command's `cwd`; +- the current process directory after another command; or +- the directory containing an executable. + +The initial structured-command surface accepts only working directories that +resolve within the effective workspace capability. Absolute paths and lexical +or symlink escapes are rejected. + +This confinement is deliberate. A later capability RFC may allow an explicit +external directory handle, but a rendered string must not silently expand the +process's ambient filesystem authority. + +## 6. Existence and type checks + +Netsuke performs all safe lexical and capability-boundary validation during +manifest compilation. + +Directory existence is checked by the action runner immediately before spawning +the execution unit. The path must identify a directory at that time. + +Runtime errors distinguish: + +- path not found; +- path exists but is not a directory; +- permission denied; +- symlink or capability escape; +- path changed between validation and spawn where the platform exposes that + distinction; and +- platform process API rejection. + +Netsuke does not create the directory automatically. + +## 7. Direct-mode semantics + +For a direct structured command, the action runner configures the process using +the platform process API equivalent of `std::process::Command::current_dir`. It +does not prefix argv with a shell command. + +Executable resolution follows RFC 0001 section 9 with this clarification: + +- a bare program name is resolved through the effective child `PATH`; +- a relative program containing a path separator is resolved against the + normalized `cwd` to an absolute, capability-checked executable path before + `current_dir` is set; and +- an absolute executable remains subject to existing capability and platform + policy. + +Arguments are unaffected by `cwd`; Netsuke does not rewrite relative path +arguments because only the called program knows their grammar. + +Example: + +```yaml +command: + invoke: cargo test --all-targets + cwd: rust_extension +``` + +This launches `cargo` with `rust_extension` as its process working directory +and passes the remaining arguments unchanged. + +## 8. Shell-mode semantics + +For `shell: true`, Netsuke sets the shell process working directory through the +process API before executing the rendered shell source. + +Example: + +```yaml +command: + invoke: printf '%s\n' "$PWD" + shell: true + cwd: backend +``` + +Netsuke must not lower this to `cd backend && ...`; doing so would reintroduce +shell-dialect quoting and error-propagation differences. + +The shell sees its ordinary working-directory variables and built-ins after +startup. Any `cd` performed inside that shell process remains local to the +execution unit. + +## 9. Command-list boundaries + +Each structured block receives its own `cwd`. The value does not carry into a +later structured block, rule reference, script item, or separate legacy shell +group. + +```yaml +command: + - invoke: cargo build + cwd: workspace-a + - invoke: cargo test + cwd: workspace-b +``` + +The second command starts directly in `workspace-b`, regardless of changes made +by the first process. + +RFC 0001 legacy compatibility remains unchanged: + +```yaml +command: + - cd workspace-a + - cargo build +``` + +This all-string list remains one legacy shell group, so the shell `cd` carries +between its entries. Netsuke does not reinterpret or migrate legacy groups +automatically. + +## 10. Pipeline semantics + +Every structured pipeline stage may specify a distinct `cwd`: + +```yaml +command: + - invoke: generator --format json + cwd: producer + pipe: true + - invoke: validator --schema schema.json + cwd: consumer + stdout: validated.json +``` + +Netsuke establishes pipes before spawning the stages and applies each stage's +working directory independently. + +A stage's `cwd` does not affect pipe bytes or the next stage's path resolution. +Spawn failure caused by one invalid `cwd` follows RFC 0001 pipeline cleanup: +already-started stages are terminated and reaped, and every failure identifies +the affected stage. + +## 11. Stream-path relationship + +RFC 0001 section 12 currently resolves `stdin`, `stdout`, `stderr`, and `tee` +paths relative to Netsuke's effective working directory after CLI `-C` +processing. + +This amendment preserves that rule. Stream paths do **not** become relative to +`cwd`. + +Example: + +```yaml +command: + invoke: cargo test + cwd: rust_extension + stdout: artefacts/rust-extension-test.log +``` + +The child runs in `rust_extension`, while the output path resolves from the +effective workspace root. + +Keeping these bases distinct has three advantages: + +- graph-facing artefact paths do not silently change when command placement + changes; +- stream collision validation remains one workspace-relative operation; and +- a reviewer can reason about generated files without mentally applying each + process directory. + +A future object-valued path syntax may allow `relative_to: cwd` explicitly, but +that is outside this amendment. + +## 12. Bundle and include relationship + +Commands originating from included fragments or bundles still resolve `cwd` +against the importing build's effective workspace root. + +A bundle must not gain ambient access to its own source directory merely +because its manifest file lives there. A reusable bundle that needs a +subproject path should accept it as a typed parameter and use that value in +`cwd`. + +The provenance record retains both: + +- the source fragment or bundle that declared `cwd`; and +- the rendered workspace-relative working directory. + +This keeps bundle execution portable across repository layouts. + +## 13. Execution IR amendment + +RFC 0001's illustrative `CommandBlock` gains: + +```rust +pub struct CommandBlock { + pub invoke: String, + pub shell: bool, + pub cwd: Option, + pub env: BTreeMap, + pub stdin: Option, + pub stdout: Option, + pub stderr: Option, + pub tee: Option, + pub pipe: PipeStream, +} +``` + +`PipeStream` is the RFC 0001 enum with `None`, `Stdout`, and `Stderr` variants. +The `pipe: true` compatibility spelling normalizes to `Stdout`; RFC 0010 +refines the `Stderr` selection with its raw-byte pipeline semantics. + +The rendered `ProcessSpec` also carries a normalized, capability-relative +working-directory value or directory handle. + +The action plan must not contain an unvalidated absolute host path when a +workspace-relative identity or leased directory capability is sufficient. + +## 14. Action runner and generated sidecars + +The action runner receives the effective workspace root through the same leased +execution context that owns action-plan sidecars. + +Before spawn it: + +1. resolves the normalized `cwd` through the workspace directory capability; +2. verifies it is still a directory; +3. configures the child process directory; +4. applies the environment overlay; +5. configures streams and pipeline handles; and +6. spawns the process. + +The order above is conceptual. Implementations may prepare handles differently, +but a failed directory resolution must occur before the child executes. + +Generated plans include enough schema versioning that a runner which does not +understand `cwd` rejects the plan rather than ignoring the field. + +## 15. Diagnostics and observability + +A `cwd` failure identifies: + +- enclosing action, target, or rule; +- command-list item and pipeline stage; +- original source file and span; +- rendered workspace-relative directory; +- failure category; and +- bundle/include provenance where applicable. + +Absolute host paths should be omitted or redacted when the relative path is +sufficient. + +Telemetry may record bounded categories such as: + +- default versus explicit working directory; +- direct versus shell mode; and +- failure category. + +Directory values must not become metric labels. + +## 16. Security properties + +The amendment provides these properties: + +- `cwd` cannot alter argv boundaries or shell syntax in direct mode; +- a rendered relative path cannot escape the workspace capability; +- one command's directory cannot mutate another command's process state; +- bundle source location does not confer filesystem authority; and +- generated plans retain a validated directory identity rather than arbitrary + shell source. + +It does not prevent the child from traversing from its working directory using +its own ambient filesystem permissions. Broader child sandboxing remains a +separate execution policy. + +## 17. Compatibility and migration + +Existing structured command proposals without `cwd` retain the effective +workspace root as their process directory. + +Legacy scalar and all-string-list commands retain their existing shell +semantics. + +Mechanical migration examples are: + +```yaml +# Before +command: cd rust_extension && cargo clippy --all-targets + +# After +command: + invoke: cargo clippy --all-targets + cwd: rust_extension +``` + +and: + +```yaml +# Before +command: + - cd backend + - cargo test + +# After +command: + invoke: cargo test + cwd: backend +``` + +Only migrate a legacy list when no other shell state must persist between its +entries. + +## 18. Validation additions + +RFC 0001 section 15 gains: + +- `cwd` must render to one non-empty scalar string; +- `cwd` must contain no NUL; +- relative path normalization must remain inside the effective workspace; +- absolute paths are invalid in the initial surface; +- bundle and fragment origin does not change the resolution base; +- each pipeline stage validates its own directory; and +- an action plan containing `cwd` requires a runner schema that supports it. + +Directory existence and type remain runtime validation as described in section +6. + +## 19. Test additions + +RFC 0001's test strategy gains: + +- direct and shell commands observing the requested directory; +- default-directory compatibility tests; +- relative executable resolution from `cwd`, including normalization of + separator-containing paths to absolute capability-checked paths before + `current_dir` is set; +- paths containing spaces and Unicode; +- lexical `..`, absolute, and symlink escape rejection; +- missing, non-directory, and permission failures; +- independent directories across adjacent structured blocks; +- distinct directories across pipeline stages; +- pipeline cleanup when a later stage has an invalid directory; +- stream paths remaining workspace-relative rather than `cwd`-relative; +- fragment and bundle declarations using parameterized directories; +- Windows drive, separator, and case behaviour; and +- action-plan schema compatibility and provenance snapshots. + +Property tests should generate bounded relative paths and assert that +normalized accepted paths remain descendants of the effective workspace root. + +## 20. Alternatives considered + +### 20.1 Require `cd` under `shell: true` + +This keeps the schema smaller but defeats direct mode, differs between shells, +and mixes directory selection with command source. Rejected. + +### 20.2 Add only rule-level working directories + +Some command sequences need different directories per stage, and a rule-level +field creates inheritance and precedence questions before the primitive exists. +Rejected as the initial surface. + +### 20.3 Resolve stream paths relative to `cwd` + +This resembles shell redirection but makes graph artefact locations depend on +process placement and complicates collision checks. Rejected for the initial +field; an explicit future path object may opt in. + +### 20.4 Resolve `cwd` relative to the declaring fragment + +That makes extracted fragments and bundles location-dependent. Rejected in +favour of the effective workspace root and typed bundle parameters. + +### 20.5 Allow unrestricted absolute paths + +This grants ambient filesystem authority through a rendered string and weakens +bundle portability. Rejected until an explicit external-directory capability is +designed. + +## 21. Recommendation + +Amend RFC 0001 to include `cwd` as a first-class structured process field. + +Working directory is part of process construction, not shell syntax. Adding it +now closes a major Makefile-migration gap while preserving RFC 0001's central +safety property: rendered values remain typed process data and are never +reparsed as command structure. diff --git a/docs/rfcs/0010-runtime-bindings-and-secure-tempdirs.md b/docs/rfcs/0010-runtime-bindings-and-secure-tempdirs.md new file mode 100644 index 000000000..a4cb5d171 --- /dev/null +++ b/docs/rfcs/0010-runtime-bindings-and-secure-tempdirs.md @@ -0,0 +1,1046 @@ +# RFC 0010: Runtime bindings and secure execution contexts + +## Preamble + +- **RFC number:** 0010 +- **Amends:** RFC 0001, Structured command blocks and argv templates +- **Also refines:** RFC 0009, Structured-command working directories +- **Status:** Proposed +- **Created:** 2026-08-26 +- **Target:** Structured command execution context and action-runner IR + +## 1. Summary + +This amendment adds four related structured-command capabilities: + +1. capture a command's standard output into a named environment binding; +2. connect one pipeline stage's standard error to the next stage's standard + input; +3. select `cwd` from a named environment binding; and +4. execute in a securely created temporary directory, optionally binding that + directory to an environment name for later commands. + +The features share one action-local runtime binding context. The runner owns +that context and passes its values to child processes without mutating +Netsuke's own process environment. + +A typical sequence is: + +```yaml +command: + - invoke: workspace-locator --relative + stdout: + env: WORKSPACE_DIR + - invoke: cargo test --all-targets + cwd: + env: WORKSPACE_DIR +``` + +A secure temporary workspace is: + +```yaml +command: + - invoke: prepare-fixture + cwd: + tempdir: + env: FIXTURE_DIR + - invoke: verify-fixture + cwd: + env: FIXTURE_DIR +``` + +A diagnostic pipeline may use standard error as its data stream: + +```yaml +command: + - invoke: compiler broken-input.rs + pipe: stderr + - invoke: diagnostic-normalizer +``` + +The central safety rules are: + +> Runtime bindings are scoped data owned by the action runner, not mutations of +> the parent process environment. +> +> Text captured from a child cannot grant filesystem authority merely by +> containing an absolute path. +> +> A runner-created temporary directory carries an explicit directory +> capability and is cleaned on every completion path. + +Upon acceptance, RFC 0001 must be read as if its stream, pipeline, environment, +validation, action-runner, security, and test sections included the semantics +below. The working-directory amendment must be read as if `cwd` accepted the +additional environment and temporary-directory forms in section 6. RFC 0009 +therefore remains the literal-path foundation for these additional forms. + +The implementation PR should fold this amendment into RFC 0001 and its +working-directory text before the RFC moves from Proposed to Accepted. + +## 2. Motivation + +Several common Make and shell patterns remain awkward even after argv-safe +structured commands gain literal `cwd` support. + +### 2.1 Capture a discovered value + +Builds often run a small discovery command and export its result: + +```sh +WORKSPACE_DIR=$(tool locate-workspace) +cd "$WORKSPACE_DIR" +cargo test +``` + +Encoding this through shell command substitution loses the direct-process +safety model. Writing the result to a file is possible but creates unnecessary +filesystem state and forces every consumer to parse the file. + +### 2.2 Parse diagnostics rather than ordinary output + +Some tools deliberately write machine-readable or normalization-worthy data to +standard error. Shell pipelines express this with implementation-specific file +descriptor syntax. A structured pipeline should state which output stream feeds +the next stage without relying on `2>&1`, process substitution, or PowerShell +redirection rules. + +### 2.3 Select a directory at runtime + +The literal `cwd` amendment covers repository-known directories but not a path +selected by a previous command, supplied through a controlled environment +binding, or created for one action execution. + +### 2.4 Use a private temporary workspace + +Build and test tasks frequently need a scratch directory for extraction, +fixture generation, signing inputs, package assembly, or untrusted intermediate +files. Calling `mktemp`, `%TEMP%`, or a PowerShell helper from shell source has +platform-specific quoting, permission, cleanup, and race behaviour. + +Temporary-directory creation is a capability and lifecycle concern. It belongs +in the action runner rather than in recipe text. + +## 3. Goals and non-goals + +### 3.1 Goals + +This amendment aims to: + +- capture bounded UTF-8 standard output into a named runtime environment + binding; +- make that binding available to later execution units in one command sequence; +- preserve child-process environment isolation; +- let `cwd` consume a named runtime or inherited environment value; +- distinguish ordinary text paths from runner-owned directory capabilities; +- securely create, expose, use, and remove temporary directories; +- support standard-error pipelines without merging streams; +- retain deterministic validation and precise source provenance; +- define failure and cleanup behaviour across commands and pipelines; and +- preserve existing RFC 0001 and legacy-command behaviour. + +### 3.2 Non-goals + +This amendment does not: + +- expose captured values to manifest-time Jinja; +- propagate a child's environment mutations back to Netsuke or later children; +- capture arbitrary binary output into environment variables; +- capture unbounded output; +- merge standard output and standard error into one pipe; +- pipe both output streams to one downstream standard input; +- make a runtime binding visible across independent Ninja edges; +- export a runtime binding as a target output or cache key; +- permit arbitrary absolute `cwd` values from inherited environment variables; +- preserve temporary directories after success or failure; +- provide a directory stack; +- infer graph inputs or outputs from captured values or temporary files; or +- replace explicit build artefacts with hidden temporary state. + +A future RFC may define typed action outputs that cross graph edges. This +amendment is intentionally limited to one action-runner sequence. + +## 4. Terminology + +- **Runtime binding:** A named value held by the action runner and injected into + later child environments. +- **Text binding:** A UTF-8 runtime binding without filesystem authority. +- **Directory binding:** A runtime binding paired with a runner-owned directory + capability. +- **Binding context:** The ordered, action-local set of runtime bindings visible + to one command sequence. +- **Producer:** A command field that creates a runtime binding. +- **Consumer:** A later command field that reads a runtime binding. +- **Secure temporary directory:** An unpredictable, owner-private directory + created and removed by the action runner through platform process and + filesystem APIs. +- **Selected pipeline stream:** The one output stream connected to the next + structured pipeline stage's standard input. + +## 5. Binding-context semantics + +### 5.1 Scope + +Every structured command sequence receives a fresh binding context when the +action runner begins the sequence. The context is destroyed when that sequence +finishes, whether by success, failure, cancellation, or spawn error. + +A sequence means the heterogeneous `command` list owned by one action, target, +or rule execution unit. A single structured command also has a context, though +a captured value has no later consumer unless a future action-output RFC uses +it. + +Bindings are visible to later items in lexical execution order. They do not +cross: + +- independent Ninja edges; +- separate actions or targets; +- rule-reference boundaries defined by RFC 0001; +- a newly invoked Netsuke process; or +- manifest evaluation. + +Script items and newly spawned legacy shell groups within the same +action-runner sequence receive the current binding context as part of their +child environment. An all-string legacy command list remains one legacy shell +group and cannot produce a structured runtime binding. + +### 5.2 Effective child environment + +For each child process, Netsuke constructs the effective environment in this +order: + +1. the runner's inherited child-environment base; +2. sequence-local runtime bindings; and +3. the structured command's explicit `env` overlay. + +Later layers win for the child process only. An explicit command `env` entry +may shadow a runtime binding for that command without changing the stored +binding. + +The parent Netsuke process environment is never modified. + +### 5.3 Names and collisions + +Runtime binding names are literal YAML strings. Jinja is not permitted in a +binding name. + +For portable behaviour, names must match: + +```text +[A-Za-z_][A-Za-z0-9_]* +``` + +Netsuke compares binding names case-insensitively on Windows and +case-sensitively on other supported platforms. The original spelling is +preserved for child environment construction and diagnostics. + +Two producers in one binding scope may not bind the same normalized name. A +producer may shadow an inherited environment variable, but the diagnostic and +structured plan record that shadowing explicitly. + +### 5.4 Commit semantics + +A producer reserves its binding name during plan validation. The value becomes +visible only after its execution unit succeeds. + +If the producer fails, is cancelled, exceeds a capture limit, emits invalid +text, or encounters a cleanup error before commit, Netsuke does not publish the +binding. + +For a structured pipeline, capability preparation is an explicit pre-spawn +phase. The runner reserves every declared directory-binding name, creates and +validates each secure temporary directory, and builds a pending capability map +before starting any stage. It atomically commits that map only if preparation +completes successfully; no stage starts, and no prepared binding becomes +visible when preparation fails. A directory binding produced by this phase is +therefore visible to every stage that consumes it, including stages that start +concurrently. Its producer-success condition is successful capability +preparation, not the eventual exit status of the stage that uses the directory. + +Text captures and other execution-produced bindings retain the ordinary +producer-success rule: they are not visible until their producer execution unit +succeeds. A captured value in a pipeline commits only after every stage has +completed successfully according to RFC 0001's pipeline policy. A stage must +not consume an execution-produced binding from another stage before that +binding has committed. + +## 6. Manifest syntax + +### 6.1 Standard output to an environment binding + +RFC 0001's `stdout` field becomes a union. Its existing path-string form +remains unchanged. The new mapping form captures standard output: + +```yaml +stdout: + env: WORKSPACE_DIR + chomp: true + max_bytes: 65536 +``` + +| Field | Type | Default | Meaning | +| ----------- | ------------------------- | -------- | -------------------------------- | +| `env` | portable environment name | required | Runtime binding to create. | +| `chomp` | Boolean | `true` | Remove one trailing line ending. | +| `max_bytes` | positive integer | `65536` | Maximum captured byte count. | + +Table 1: Standard-output environment capture fields. + +Unknown keys are errors. The initial manifest-format maximum for `max_bytes` is +16 MiB. A smaller implementation hard limit is invalid because the manifest +contract must be portable across runners. + +`chomp: true` removes at most one final line ending: + +- `\r\n` is removed as one line ending; +- otherwise one final `\n` is removed; and +- all other bytes are preserved. + +It does not trim spaces, tabs, additional blank lines, or an isolated final +carriage return. + +Example: + +```yaml +command: + - invoke: printf-workspace-path + stdout: + env: WORKSPACE_DIR + - invoke: cargo check --all-targets + cwd: + env: WORKSPACE_DIR +``` + +### 6.2 Pipeline source selection + +RFC 0001's Boolean `pipe` field becomes a backwards-compatible union: + +```yaml +pipe: false +pipe: true +pipe: stdout +pipe: stderr +``` + +The meanings are: + +- `false`: no pipe to the next stage; +- `true`: compatibility spelling for `stdout`; +- `stdout`: connect this stage's standard output to the next stage's standard + input; and +- `stderr`: connect this stage's standard error to the next stage's standard + input. + +“Standard error to standard input” means the current stage's writable standard +error stream becomes the immediately following stage's readable standard input. +It never attaches a process's own writable standard error handle to its own +read-only standard input handle. + +Example: + +```yaml +command: + - invoke: compiler invalid-source.rs + pipe: stderr + - invoke: normalize-diagnostics + stdout: normalized-errors.txt +``` + +Merged standard output and standard error remain outside RFC 0001's initial +surface. + +### 6.3 Working directory from an environment name + +The `cwd` field gains a mapping form: + +```yaml +cwd: + env: WORKSPACE_DIR +``` + +The mapping contains exactly one `env` key. The name follows section 5.3. + +The runner resolves the name from the effective child environment described in +section 5.2. A sequence-local directory binding retains its directory +capability. An ordinary inherited, captured, or explicitly overlaid value is a +text binding and receives the path validation in section 9. + +### 6.4 Secure temporary working directory + +The `cwd` field also gains a temporary-directory form: + +```yaml +cwd: + tempdir: {} +``` + +This creates a private temporary directory for one execution unit, sets the +child process working directory to it, and removes it after that unit finishes. +The empty mapping is required so a literal repository directory named `tempdir` +remains unambiguous. + +A named form publishes a directory binding: + +```yaml +cwd: + tempdir: + env: FIXTURE_DIR +``` + +When `env` is present: + +- the runner creates the directory before the execution unit starts; +- the current child receives the path under `FIXTURE_DIR`; +- the command runs inside that directory; +- later execution units in the same sequence inherit the binding; +- `cwd: { env: FIXTURE_DIR }` consumes the retained directory capability; and +- the runner removes the directory when the entire sequence finishes. + +The `tempdir` mapping accepts exactly one optional field: + +| Field | Type | Default | Meaning | +| ----- | ------------------------- | ------- | ------------------------------------------- | +| `env` | portable environment name | absent | Publish a sequence-local directory binding. | + +Table 2: Secure temporary-directory fields. + +Retention, custom prefixes, user-selected roots, and keep-on-failure modes are +not part of the initial surface. Security-sensitive temporary data is removed +on every completion path by default. + +## 7. Standard-output capture semantics + +### 7.1 Byte collection + +The action runner drains the child's standard output continuously to avoid pipe +back-pressure. It counts raw bytes before UTF-8 decoding or line-ending removal. + +If the byte count would exceed `max_bytes`, the runner: + +1. stops accepting additional bytes; +2. terminates the execution unit using RFC 0001's bounded termination policy; +3. reaps every affected process; +4. reports `stdout_capture_limit_exceeded`; and +5. does not publish the binding. + +The runner must not retain unbounded data while waiting for a child to exit. + +### 7.2 Text conversion + +After successful process completion, Netsuke decodes the captured bytes as +strict UTF-8. Invalid UTF-8 and embedded NUL produce typed errors and prevent +binding commit. + +The initial surface has no locale-dependent decoding and no lossy conversion. A +command that needs binary capture must write a declared file artefact instead. + +After optional `chomp`, the resulting string may be empty and may contain +embedded newlines. It is inserted into later child environments exactly as a +text value. + +### 7.3 Visibility and redaction + +Human and JSON diagnostics may expose: + +- binding name; +- capture limit; +- captured byte count; +- whether one line ending was removed; and +- success or failure category. + +They must not expose the captured value by default. Captured output may contain +credentials, tokens, paths, or user data. It must not become a metric label, +span name, cache key, or generated Ninja comment. + +### 7.4 Stream conflicts + +A standard-output environment capture is invalid when the same stage also: + +- pipes standard output to the next stage; +- redirects standard output to a path; +- tees standard output under RFC 0001; or +- requests another standard-output sink. + +It may coexist with `pipe: stderr`, because the selected pipeline stream is +standard error and standard output remains available for capture. + +In a pipeline, only a stage whose standard output is not selected for a later +stage may capture that output. The binding commits after complete pipeline +success. + +## 8. Standard-error pipeline semantics + +### 8.1 Raw stream behaviour + +`pipe: stderr` transfers raw bytes. Netsuke does not decode, normalize, frame, +or add prefixes to the stream. + +The current stage's standard output follows its independently configured sink: +inheritance, file redirection, teeing, or environment capture. + +The current stage's standard error is consumed by the pipe and is not inherited +by the terminal. A future tee-to-pipe surface may duplicate it explicitly, but +implicit duplication is not permitted. + +### 8.2 Validation + +`pipe: stderr` is invalid when: + +- the stage is the final structured stage in a pipeline; +- the next item is a rule reference, script item, or legacy shell boundary; +- the next stage specifies its own `stdin` source; +- the current stage specifies a standard-error path or another standard-error + sink; or +- the current stage also requests standard-output piping. + +Exactly one predecessor stream may feed one stage's standard input. + +### 8.3 Exit and cleanup behaviour + +RFC 0001's pipeline exit policy remains unchanged. Selecting standard error as +the data stream does not turn a non-zero producer status into success. + +When a stage cannot spawn or the pipeline is cancelled, Netsuke closes every +pipe endpoint, terminates started children, drains or abandons streams +according to the bounded cleanup policy, and reaps all processes. + +Diagnostics identify the selected stream and the source and destination stages. + +## 9. Environment-selected working directories + +### 9.1 Resolution order + +For `cwd: { env: NAME }`, Netsuke reads `NAME` after applying the environment +precedence in section 5.2. + +The value must be present, non-empty, valid UTF-8, and free of NUL. + +A relative text value is resolved against Netsuke's effective working directory +after CLI `-C`, exactly like the literal path form in the working-directory +amendment. + +### 9.2 Text values and filesystem authority + +An ordinary text value receives the same confinement as a literal `cwd`: + +- lexical normalization must remain inside the effective workspace; +- symlink resolution must not escape the workspace capability; +- the path must identify a directory at spawn time; and +- an absolute path is rejected in the initial surface. + +This rule applies whether the text originated from: + +- the inherited environment; +- an explicit command `env` overlay; or +- standard-output capture. + +A child cannot gain external-directory authority merely by printing an absolute +path and having it captured into a binding. + +### 9.3 Directory bindings + +A secure temporary-directory producer creates a typed directory binding. The +binding contains: + +- the child-visible path string; +- a runner-owned directory capability; +- lifecycle ownership; and +- producer provenance. + +`cwd: { env: NAME }` may use that capability even when the temporary directory +lives outside the workspace. The permission does not derive from the path +string and cannot be forged by a text binding with the same spelling. + +If a per-command `env` overlay shadows a directory binding's name, the value is +resolved as ordinary text for that command and the external directory +capability is not inherited through the shadow. + +### 9.4 Executable and argument behaviour + +After resolving the directory, direct and shell execution follow RFC 0009: + +- bare executable names use the effective child `PATH`; +- relative executable paths containing a separator resolve from `cwd`; +- arguments are passed unchanged; and +- shell mode receives the directory through the process API, not generated + `cd` source. + +## 10. Secure temporary-directory semantics + +### 10.1 Root selection + +The runner selects its temporary root once through application configuration or +the platform secure-temporary-directory API at the composition boundary. + +A command's `env` overlay, runtime bindings, or captured output cannot redirect +temporary-directory creation to another root. + +The root is opened and retained through a directory capability before creating +an execution directory. + +### 10.2 Creation + +The runner creates each temporary directory atomically with an unpredictable +cryptographic name and no pre-existing path reuse. + +On Unix-like platforms, the directory is owner-only, equivalent to mode `0700`, +before the path becomes visible to a child. On Windows, the directory receives +an access-control list restricted to the current security principal and +required system access. + +The implementation must not create a world-readable directory and tighten it +afterwards. + +Netsuke rejects a temporary root or selected directory that resolves through an +unexpected symlink or non-directory object. + +### 10.3 Binding and pipeline preparation + +A command-scoped temporary directory is created immediately before its +execution unit and removed immediately after that unit has been reaped and all +stream tasks have completed. + +A named sequence-scoped temporary directory is created when its producer is +prepared and removed after the complete sequence finishes. + +For a structured pipeline, the runner materializes every secure temporary +directory and directory binding required by any stage before spawning the first +stage. This lets one stage consume a named directory binding created by another +stage's declaration without introducing a race between concurrent process +starts. + +If preparation of any directory fails, no pipeline stage starts. + +### 10.4 Child environment + +When the tempdir form contains `env`, the current and later children receive +the absolute child-visible path under that environment name. + +Netsuke does not implicitly rewrite `TMPDIR`, `TMP`, `TEMP`, `HOME`, or any +other conventional variable. A manifest requiring those values must set them +explicitly through `env`, preferably to the named binding once runtime binding +references are supported in environment overlays by a later amendment. + +The child always starts with its process working directory set to the secure +temporary directory. + +### 10.5 Cleanup + +Cleanup runs after success, failure, cancellation, timeout, signal, spawn +error, and pipeline setup failure. + +The runner: + +1. waits for or terminates every owned child process; +2. closes stream tasks and handles; +3. recursively removes entries through the retained directory capability; +4. does not follow symlinks during removal; and +5. removes the root execution directory last. + +A cleanup failure is not silently ignored: + +- after otherwise successful execution, cleanup failure makes the execution + unit fail; +- after an existing execution failure, cleanup failure is attached as a + secondary cause; and +- structured output reports both categories without exposing sensitive file + names unnecessarily. + +The initial surface does not retain failed temporary directories for debugging. +A future retention feature must be explicit, security-reviewed, and disabled by +default. + +### 10.6 Background descendants + +Netsuke owns only processes created through the structured execution unit and +its documented process-group or job-object policy. A child that deliberately +detaches an untracked descendant may keep files or handles alive and cause +cleanup failure. + +The diagnostic must state that detached descendants are incompatible with +secure temporary-directory cleanup unless a future supervised-background-task +model owns them. + +## 11. Interaction examples + +### 11.1 Capture a repository-relative workspace path + +```yaml +command: + - invoke: project-tool print-build-root --relative + stdout: + env: BUILD_ROOT + - invoke: cargo clippy --all-targets + cwd: + env: BUILD_ROOT +``` + +The captured value is text. It may select a directory inside the effective +workspace but cannot escape it. + +### 11.2 Share a secure fixture directory + +```yaml +command: + - invoke: fixture-generator + cwd: + tempdir: + env: FIXTURE_DIR + - invoke: fixture-validator . + cwd: + env: FIXTURE_DIR +``` + +`FIXTURE_DIR` is both an environment value and a typed directory capability. +The directory survives between the two commands and is removed after the +sequence. + +### 11.3 Capture stdout while piping stderr + +```yaml +command: + - invoke: compiler source.rs + stdout: + env: COMPILER_SUMMARY + pipe: stderr + - invoke: diagnostic-normalizer +``` + +The compiler's standard error feeds the normalizer. Its standard output is +captured separately and commits only if the complete pipeline succeeds. + +### 11.4 Reject an untrusted absolute path + +```yaml +command: + - invoke: untrusted-tool print-directory + stdout: + env: DIRECTORY + - invoke: inspect + cwd: + env: DIRECTORY +``` + +If `DIRECTORY` contains `/tmp/outside` or an absolute Windows path, the second +command fails before spawn because the text binding carries no external +directory capability. + +## 12. Execution IR amendment + +The execution IR gains typed stream, binding, and directory selectors. One +illustrative shape is: + +```rust +pub enum PipeStream { + None, + Stdout, + Stderr, +} + +pub struct StdoutEnvCapture { + pub name: EnvName, + pub chomp: bool, + pub max_bytes: usize, +} + +pub enum StdoutSink { + Inherit, + File(Utf8PathBuf), + Tee(Utf8PathBuf), + Environment(StdoutEnvCapture), + Pipe, +} + +pub enum WorkingDirectory { + WorkspaceRoot, + WorkspacePath(Utf8PathBuf), + Environment(EnvName), + SecureTempdir { + binding: Option, + }, +} + +pub struct RuntimeBindingContext { + pub values: BTreeMap, +} + +pub enum BindingValue { + Text(String), + Directory(DirectoryBinding), +} +``` + +The exact Rust names may change, but the distinctions are normative: + +- selected pipeline stream is not a Boolean in the IR; +- environment capture is a bounded text sink; +- `cwd` text and directory capabilities remain different variants; and +- the binding context is action-runner state, not a process-global environment + map. + +Generated action plans reserve producer names and include source provenance, +limits, selectors, and lifecycle scope. They never include runtime-captured +values or runtime-generated temporary paths. + +A runner that does not understand these plan variants must reject the plan +schema rather than ignoring them. + +## 13. Validation additions + +Manifest compilation rejects: + +- unknown capture, pipe, `cwd`, or tempdir keys; +- invalid or dynamic environment names; +- duplicate normalized producer names in one binding scope; +- a pipeline stage that consumes an execution-produced binding from another + stage because that binding cannot commit before pipeline spawn; +- zero or excessive capture limits; +- standard-output capture combined with another standard-output sink; +- `pipe: stderr` combined with another standard-error sink; +- any pipe on the terminal stage; +- a downstream stage with both a predecessor pipe and explicit `stdin`; +- two predecessor streams targeting one standard input; +- a pipeline crossing a rule, script, or legacy boundary; +- a tempdir mapping containing fields other than optional `env`; and +- an action plan whose runner schema cannot represent the required variants. + +Runtime validation rejects: + +- missing or empty environment-selected directories; +- invalid UTF-8 or NUL in captured values or directory variables; +- workspace escape from text-selected `cwd`; +- missing or non-directory paths; +- capture overflow; +- invalid captured UTF-8; +- secure temporary-directory creation or permission failure; +- incomplete secure cleanup; and +- a failed pre-spawn capability-preparation commit, which must leave every + prepared directory binding unpublished and start no pipeline stage. + +## 14. Diagnostics and observability + +Every failure identifies: + +- enclosing action, target, or rule; +- command-list item and pipeline stage; +- source file and span; +- selected stream or working-directory source; +- binding name where applicable; +- failure category; and +- include or bundle provenance. + +Human output may display a workspace-relative path when useful. +Runtime-generated secure temporary paths and captured values are redacted by +default. + +Bounded telemetry may record: + +- capture requested and result category; +- size bucket, not exact content; +- selected pipeline stream; +- literal, environment, or secure-tempdir `cwd` source; +- command-scoped or sequence-scoped tempdir lifetime; and +- cleanup result category. + +Binding names, values, arbitrary paths, tag names, and command output must not +be metric labels. + +## 15. Security properties + +This amendment provides these properties: + +- stdout capture cannot change argv or manifest structure; +- captured text does not mutate Netsuke's parent environment; +- capture is bounded before decoding; +- captured text cannot manufacture an external directory capability; +- stderr piping uses owned process handles rather than shell descriptor syntax; +- secure tempdirs are private before child access; +- temporary paths are unpredictable and never reused deliberately; +- cleanup uses retained directory authority and does not follow symlinks; and +- runtime-generated values do not enter graph fingerprints or metadata by + accident. + +The amendment does not make executed programs trustworthy. A child retains its +normal process and filesystem authority beyond the selected working directory +unless another sandbox policy restricts it. + +## 16. Compatibility and migration + +Existing RFC 0001 forms remain valid: + +- `stdout: path` retains file-redirection semantics; +- `pipe: false` and `pipe: true` retain their meanings, with `true` normalized + to `stdout`; +- literal string `cwd` retains RFC 0009's semantics; and +- legacy command strings and command-string lists remain unchanged. + +Mechanical migrations include: + +```yaml +# Before +command: WORKSPACE_DIR=$(tool locate) && cd "$WORKSPACE_DIR" && cargo test + +# After +command: + - invoke: tool locate --relative + stdout: + env: WORKSPACE_DIR + - invoke: cargo test + cwd: + env: WORKSPACE_DIR +``` + +and: + +```yaml +# Before +command: compiler source.rs 2>&1 | normalizer + +# After, when diagnostics are written only to stderr +command: + - invoke: compiler source.rs + pipe: stderr + - invoke: normalizer +``` + +The latter is not equivalent to merged `2>&1`; standard output remains a +separate stream. A manifest requiring merged streams must retain explicit shell +mode until a later RFC defines it. + +## 17. Test strategy + +The implementation must include: + +### Standard-output capture + +- default one-line-ending chomp and `chomp: false` preservation; +- empty output; +- embedded newlines; +- CRLF and LF endings; +- invalid UTF-8 and NUL rejection; +- exact limit, one byte over limit, and bounded process termination; +- value redaction in human and JSON diagnostics; +- binding commit only after success; +- duplicate producer-name rejection; and +- Windows case-insensitive name collisions. + +### Environment-selected `cwd` + +- inherited, captured, explicit-overlay, and directory-binding sources; +- precedence between sequence bindings and command overlays; +- relative workspace path success; +- absolute and symlink escape rejection for text values; +- absolute secure-tempdir capability success; +- missing, empty, non-directory, invalid UTF-8, and NUL failures; and +- relative executable resolution from the selected directory. + +### Standard-error pipelines + +- raw byte preservation; +- standard output remaining independent; +- conflict with standard-error file redirection; +- conflict with downstream explicit standard input; +- final-stage rejection; +- producer and consumer failure propagation; +- cancellation and reaping; and +- capture-stdout plus pipe-stderr coexistence. + +### Secure temporary directories + +- owner-only Unix permissions before child access; +- restricted Windows access control; +- unpredictable unique names under concurrency; +- command-scoped and sequence-scoped lifetime; +- use across adjacent commands and pipeline stages; +- creation failure before any pipeline spawn; +- atomic pre-spawn capability preparation and commit for all pipeline directory + bindings; +- every consuming pipeline stage seeing a directory binding after successful + preparation; +- no stage starting, and no directory binding becoming visible, after + preparation failure; +- cleanup after success, failure, cancellation, timeout, and spawn error; +- no symlink following during recursive removal; +- cleanup failure as primary or secondary error; +- detached descendant or open-handle failure diagnostics; and +- no leakage of absolute temporary paths into plans, snapshots, or metrics. + +Property tests should generate bounded command sequences and assert: + +- a binding is visible only after its producer commits; +- a prepared directory binding commits only after the complete capability + preparation phase succeeds; +- an execution-produced binding is not visible to another pipeline stage + before its producer succeeds; +- no text binding acquires directory capability; +- every created secure tempdir has exactly one cleanup owner; and +- pipeline stream selection has at most one source for each downstream stdin. + +A bounded Kani harness may model binding reservation, commit, command failure, +and cleanup state transitions. + +## 18. Alternatives considered + +### 18.1 Shell command substitution + +This is concise but reintroduces shell parsing, platform differences, and +unbounded implicit capture. Rejected for structured mode. + +### 18.2 Write every discovered value to a file + +Files are appropriate for declared artefacts and binary data, but excessive for +small action-local text values. Rejected as the only mechanism. + +### 18.3 Allow captured output in manifest-time Jinja + +Runtime values do not exist when the static Ninja graph is compiled. Rejected. + +### 18.4 Merge stdout and stderr by default + +Merging destroys stream identity and changes ordering semantics. Rejected; this +amendment selects exactly one pipeline source. + +### 18.5 Treat every environment-selected absolute path as trusted + +Environment variables are mutable ambient strings and captured output is +untrusted child data. Rejected in favour of typed directory bindings. + +### 18.6 Implement secure tempdirs through `mktemp` + +That depends on shell availability, command variants, umask, quoting, output +capture, and manual cleanup. Rejected in favour of runner-owned filesystem +operations. + +### 18.7 Keep failed tempdirs automatically + +This is convenient for debugging but leaks potentially sensitive inputs and +makes cleanup non-deterministic. Rejected from the initial surface. + +## 19. Open questions + +- Should a future field expose a captured runtime binding directly as one argv + element without routing through the child environment? +- Should environment overlays later accept a typed reference to an existing + runtime binding rather than only compile-time strings? +- Should secure tempdirs support an explicit capability-scoped root selected by + repository policy? +- Should a future debug mode retain a failed tempdir after an explicit consent + gate and print its path only to a protected diagnostic sink? +- Should merged stdout and stderr become a separate pipeline source, or remain + an explicit shell-only operation? + +These questions do not alter the initial requirements for bounded capture, +typed directory authority, parent-environment isolation, or mandatory cleanup. + +## 20. Recommendation + +Amend RFC 0001 with action-local runtime bindings, bounded +stdout-to-environment capture, explicit standard-error pipelines, +environment-selected working directories, and runner-owned secure temporary +directories. + +Together, these features replace several high-value shell idioms without +weakening the structured-command trust boundary. They also create a coherent +execution context for downstream Netsukefile migration: values may flow between +ordered commands, but they remain typed, scoped, bounded, and incapable of +silently granting authority.