diff --git a/CHANGELOG.md b/CHANGELOG.md index 86d6026f..9859da9a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -31,6 +31,11 @@ * Added first-class remote reboot support. * Expanded built-in provisioning namespaces and metrics. * Added schema hashing for provisioning schema caching. +* Added per-namespace schema hashes and repurposed `GET_CAPABILITIES` to return a namespace hierarchy map, letting clients fetch schema lazily per namespace. +* Added `NamespaceFilter` support to `GET_SCHEMA` so clients can fetch only the namespaces they don't already have cached. +* Added `Draft` flag to `GET_STATE`: when `true`, the response includes drafts alongside working values in a single round-trip. Response body is now `{Values, Drafts?, Hash}`. +* Added `PriorHash` cache short-circuit to `GET_STATE`: clients echo a previously-received `Hash`, and the server responds with `{Unchanged: true}` when the state would be byte-identical. +* Added `IncludeState` flag to `SET_STATE` and `COMMIT`: when `true`, the response includes post-op `PostOpValues` / `PostOpDrafts` / `PostOpHash`, eliminating the follow-up `GET_STATE` round-trip after a save or commit. * Added provisioning payload compression support. * Embedded Heatshrink compression implementation. @@ -127,6 +132,12 @@ * Persisted path-table data created by previous releases may not be reusable after upgrade and may be rebuilt automatically from network announces. * Applications using internal persistence formats should verify compatibility. +* **Provisioning wire format changed.** Existing v0.4-era clients will not interoperate with a v0.5 server: + * `GET_CAPABILITIES` now returns an array of namespace map entries (`{NsId, NsName, NsParent, NsFieldCount, NsSchemaHash}`), replacing the previous bare array of ids. + * `GET_STATE` responses are wrapped in `{Values, Drafts?, Hash}` (previously the response body was the `{ns_id: {fid: value}}` map directly). The old `Pending` request flag is replaced by `Draft`, with new "include drafts alongside working" semantics. + * `SET_STATE` requests use a new envelope `{State: {...}, IncludeState?, ReqCompress?}` (previously the top-level map was the state map itself). + * `COMMIT` requests use a map envelope `{NamespaceFilter?, IncludeState?, ReqCompress?}` instead of a bare array of ns ids. + * The `Pending` request key and `Applied`-slot-3 collision in the old commit response are gone. See [`docs/provisioning_client_guide.md`](docs/provisioning_client_guide.md) for the current wire format. * Existing applications using function-pointer callback handlers do not require immediate changes but should migrate toward std::function callbacks. ## Contributors diff --git a/README.md b/README.md index de5a251b..96629e3c 100644 --- a/README.md +++ b/README.md @@ -117,7 +117,7 @@ RNS::Bytes response = RNS::Provisioning::Provisioner::instance().handle_message( send_to_transport(response); ``` -Supported operations are `GET_SCHEMA`, `GET_INFO`, `GET_CAPABILITIES`, `GET_STATE`, `SET_STATE`, `COMMIT`, `DISCARD`, and `FACTORY_RESET`. The envelope is a 3-element MsgPack array `[op_id, seq, payload]`; see `src/microReticulum/Provisioning/Ops.h` for op-id and key constants. +Supported operations are `GET_SCHEMA`, `GET_INFO`, `GET_CAPABILITIES`, `GET_STATE`, `SET_STATE`, `COMMIT`, `DISCARD`, `FACTORY_RESET`, and `REBOOT`. The envelope is a 3-element MsgPack array `[op_id, seq, payload]`; see `src/microReticulum/Provisioning/Ops.h` for op-id and key constants, and [`docs/provisioning_client_guide.md`](docs/provisioning_client_guide.md) for the full wire format (per-op request/response shapes, `Draft`/`IncludeState`/`PriorHash` round-trip optimizations, and heatshrink-compressed responses). ### Manual operations (no wire) diff --git a/docs/provisioning_client_guide.md b/docs/provisioning_client_guide.md index 14e39831..fcd3cd13 100644 --- a/docs/provisioning_client_guide.md +++ b/docs/provisioning_client_guide.md @@ -19,15 +19,19 @@ This guide is for developers building **clients** that talk to a microReticulum - [`COMMIT`](#commit-op--6) - [`DISCARD`](#discard-op--7) - [`FACTORY_RESET`](#factory_reset-op--8) + - [`REBOOT`](#reboot-op--9) - [`ERROR` response](#error-response-op--101) -6. [Field types](#field-types) -7. [Field flags](#field-flags) -8. [Field schema entry](#field-schema-entry) -9. [Hierarchical namespaces](#hierarchical-namespaces) -10. [Error code reference](#error-code-reference) -11. [Built-in namespaces and fields](#built-in-namespaces-and-fields) -12. [Recommended client lifecycle](#recommended-client-lifecycle) -13. [Wire wire-format quick reference](#wire-format-quick-reference) +6. [PriorHash cache short-circuit](#priorhash-cache-short-circuit) +7. [IncludeState round-trip elimination](#includestate-round-trip-elimination) +8. [Response compression](#response-compression) +9. [Field types](#field-types) +10. [Field flags](#field-flags) +11. [Field schema entry](#field-schema-entry) +12. [Hierarchical namespaces](#hierarchical-namespaces) +13. [Error code reference](#error-code-reference) +14. [Built-in namespaces and fields](#built-in-namespaces-and-fields) +15. [Recommended client lifecycle](#recommended-client-lifecycle) +16. [Wire format quick reference](#wire-format-quick-reference) --- @@ -68,6 +72,7 @@ If the device receives a request before `Provisioner::begin()` has run (the firm | `COMMIT` | 6 | request → response | | `DISCARD` | 7 | request → response | | `FACTORY_RESET` | 8 | request → response | +| `REBOOT` | 9 | request → response | | `ACK` | 100 | (reserved) | | `ERROR` | 101 | response only | @@ -84,13 +89,15 @@ The schema version is reported by `GET_INFO` (key `SchemaVersion`, see below). C Clients should branch on `SchemaVersion` from `GET_INFO`. A v1-only client that reads only the first three array elements of a v2 response will still get id/name/fields and can render a flat list (just without the hierarchy). +`SchemaVersion` describes the `GET_SCHEMA` response layout only. The request/response shapes of the other ops (`GET_STATE`, `SET_STATE`, `COMMIT`, `GET_CAPABILITIES`) are documented per-op below. + ## Operation reference ### `GET_INFO` (op = 2) Probe device version and reboot state. Always safe to call before anything else. -**Request payload**: `nil`. +**Request payload**: `nil`, or an options map with `ReqCompress` (see [Response compression](#response-compression)). **Response payload**: a map. @@ -99,24 +106,49 @@ Probe device version and reboot state. Always safe to call before anything else. | 1 (`FirmwareVersion`) | `str` | Free-form firmware identifier (currently always `"microReticulum"` from the library; firmware can override). | | 2 (`SchemaVersion`) | `uint16` | See [Schema versioning](#schema-versioning). | | 3 (`NeedsRebootInfo`) | `bool` | `true` if the device has committed any `FF_REBOOT_REQUIRED` field since boot. Sticky until reboot or factory reset. | +| 4 (`SchemaHash`) | `uint32` | CRC32 over the serialized `GET_SCHEMA` response bytes. Clients cache the schema keyed by this hash so subsequent connections skip the full fetch when the device's schema hasn't changed. | ### `GET_CAPABILITIES` (op = 3) -List the namespace ids that the device exposes. Use this to detect optional/extended namespaces without fetching the full schema. +Fetch the namespace hierarchy map — one entry per namespace, with just enough metadata for the client to render the top-level navigation and decide which schemas need to be fetched. Combined with `GET_SCHEMA`'s namespace-filter support, this powers lazy per-namespace schema loading. + +**Request payload**: `nil`, or an options map with `ReqCompress`. + +**Response payload**: a MsgPack array of per-namespace map entries. + +Each entry is a map: + +| Key | Value type | Meaning | +|---|---|---| +| 1 (`NsId`) | `uint16` | Canonical namespace id. | +| 2 (`NsName`) | `str` | Human-readable name (may be empty). | +| 4 (`NsParent`) | `uint16` | Parent namespace id, or `0` for root. | +| 5 (`NsFieldCount`) | `uint64` | Number of fields the namespace exposes. Handy for UI placeholders. | +| 6 (`NsSchemaHash`) | `uint32` | CRC32 over just this namespace's `GET_SCHEMA` entry. Lets clients cache each namespace's schema independently. | -**Request payload**: `nil`. +Namespaces appear in registration order (same order as they'd appear in `GET_SCHEMA`). -**Response payload**: an array of `uint16` namespace ids, in registration order. +Example: ``` -[ 1, 2, 100, 101, 102 ] +[ + { 1:1, 2:"Reticulum", 4:0, 5:9, 6:0xa3f2c8d1 }, + { 1:2, 2:"Transport", 4:0, 5:5, 6:0x1e5b90c4 }, + { 1:100, 2:"Interfaces", 4:0, 5:0, 6:0x00000000 }, + { 1:101, 2:"LoRa", 4:100, 5:12, 6:0x7f2a1e63 } +] ``` ### `GET_SCHEMA` (op = 1) -Fetch the full field schema. Clients call this once per session (or whenever the schema version changes) to build a UI. +Fetch the full field schema. Clients typically call this once per session (or per changed namespace, when combined with `GET_CAPABILITIES`) to build a UI. -**Request payload**: `nil`. (Future versions may add a namespace filter.) +**Request payload**: `nil` for a full-schema fetch, or an options map: + +| Key | Value type | Meaning | +|---|---|---| +| 1 (`NamespaceFilter`) | array of `uint16` | Only return schema entries for these namespace ids. If absent, returns all namespaces. | +| 100 (`ReqCompress`) | `bool` | See [Response compression](#response-compression). | **Response payload**: a MsgPack array of **namespace entries**. Each entry is a 4-element array: @@ -145,48 +177,75 @@ Example (truncated): Read current values. Excludes [`FF_SECRET`](#field-flags) and [`FF_WRITE_ONLY`](#field-flags) fields. -**Request payload**: optional map. If present: +**Request payload**: `nil` for "all namespaces, working only," or an options map: | Key | Value type | Meaning | |---|---|---| | 1 (`NamespaceFilter`) | array of `uint16` | Only return state for these namespace ids. If absent, returns all namespaces. | -| 2 (`Pending`) | `bool` | If `true`, returned values overlay any uncommitted draft on top of working state. Default `false` (committed state only). | +| 2 (`Draft`) | `bool` | If `true`, the server includes drafts alongside working in the same response. Draft entries are sparse — only namespaces/fields that actually hold drafts appear. Default `false` (working only). | +| 4 (`PriorHash`) | `uint32` | Optional CRC32 the client received from a previous `GET_STATE` response for the same scope. Enables the server-side cache short-circuit; see [PriorHash cache short-circuit](#priorhash-cache-short-circuit). | +| 100 (`ReqCompress`) | `bool` | See [Response compression](#response-compression). | -If the payload is absent or `nil`, the response covers every namespace, with no draft overlay. +**Response payload**: a map. On a normal (cache-miss) response: -**Response payload**: a MsgPack map keyed by `uint16` namespace id, each value being a map keyed by `uint16` field id, each value being the field's typed value. +| Key | Value type | Meaning | +|---|---|---| +| 1 (`Values`) | map | `{ ns_id: { field_id: value, ... }, ... }` — the working state. | +| 2 (`Drafts`) | map (optional) | Sparse `{ ns_id: { field_id: draft_value, ... }, ... }`. Only present when the request set `Draft: true` **and** at least one in-scope namespace has drafts. | +| 3 (`Hash`) | `uint32` | CRC32 over the content that would be returned for this exact request/scope (Values + Drafts, excluding the Hash entry itself). Client can echo this back as `PriorHash` on the next request. | + +On a cache hit (client's `PriorHash` matched): + +| Key | Value type | Meaning | +|---|---|---| +| 4 (`Unchanged`) | `bool` | Always `true`. No `Values`/`Drafts`/`Hash` accompany this response — the client should reuse the cached body it received the last time it saw this Hash. | + +Response state is **flat by namespace id** even when namespaces are hierarchical — see [Hierarchical namespaces](#hierarchical-namespaces). Each namespace's fields appear under its own id; the parent doesn't contain the children's fields. + +Namespaces that have only secret/write-only/none-valued fields are omitted from `Values`. + +Example (working + drafts response): ``` { - 1: { # Reticulum namespace - 1: true, # transport_enabled - 6: 300, # persist_interval - 7: 900 # clean_interval + 1: { # Values + 1: { 1: true, 6: 300 }, # Reticulum + 101: { 1: 915000000.0 } # LoRa (child ns) }, - 101: { # LoRa namespace - 1: 915000000.0 # frequency - } + 2: { # Drafts (sparse) + 1: { 6: 600 } # only Reticulum.persist_interval has a draft + }, + 3: 0x1e5b90c4 # Hash } ``` -State is **flat by namespace id** even when namespaces are hierarchical — see [Hierarchical namespaces](#hierarchical-namespaces). Each namespace's fields appear under its own id; the parent doesn't contain the children's fields in the response. +Example (cache hit): -Namespaces that have only secret/write-only/none-valued fields are omitted from the response. +``` +{ 4: true } +``` ### `SET_STATE` (op = 5) Stage one or more field changes into the device's **draft layer**. Drafts are RAM-only — they don't take effect on the runtime, don't get persisted, and disappear on reboot unless [committed](#commit-op--6). -**Request payload**: a map keyed by `uint16` namespace id, each value a map keyed by `uint16` field id, each value the field's typed value. +**Request payload**: a request envelope. + +| Key | Value type | Meaning | +|---|---|---| +| 3 (`State`) | map | Required. `{ ns_id: { field_id: value, ... }, ... }` — the fields to stage. | +| 5 (`IncludeState`) | `bool` | Optional. If `true`, the response includes `PostOpValues` + `PostOpDrafts` + `PostOpHash` for the namespaces touched by the request. See [IncludeState round-trip elimination](#includestate-round-trip-elimination). | +| 100 (`ReqCompress`) | `bool` | See [Response compression](#response-compression). | + +Example request payload: ``` { - 1: { # Reticulum - 1: false # transport_enabled = false (draft) + 3: { # State + 1: { 1: false }, # Reticulum.transport_enabled = false + 101: { 1: 868000000.0 } # LoRa.frequency = 868 MHz }, - 101: { # LoRa - 1: 868000000.0 # frequency draft - } + 5: true # IncludeState — return updated values in the response } ``` @@ -196,11 +255,14 @@ Every value is validated against the field's declared [type](#field-types) and [ | Key | Value type | Meaning | |---|---|---| -| 1 (`Applied`) | `uint64` | Count of fields successfully staged into draft. | -| 2 (`DraftHasReboot`) | `bool` | `true` if the current draft (across all namespaces) touches any `FF_REBOOT_REQUIRED` field. UI affordance: show "you'll need to reboot to apply these changes". | -| 3 (`FieldErrors`) | array (optional, only present on partial failure) | One entry per rejected field. | +| 1 (`Applied`) | `uint64` | Count of fields successfully staged into draft. | +| 2 (`DraftHasReboot`) | `bool` | `true` if the current draft (across all namespaces) touches any `FF_REBOOT_REQUIRED` field. UI affordance: show "you'll need to reboot to apply these changes". | +| 3 (`FieldErrors`) | array (optional, only present on partial failure) | One entry per rejected field. | +| 4 (`PostOpValues`) | map (optional, only when `IncludeState: true`) | `{ ns_id: { field_id: value, ... }, ... }` — the working state for the touched namespaces. Working is unchanged by `SET_STATE`; the map is returned so the client can refresh its cache in one round-trip. | +| 5 (`PostOpDrafts`) | map (optional, only when `IncludeState: true` **and** at least one touched namespace has drafts) | Sparse `{ ns_id: { field_id: draft_value, ... }, ... }`. Reflects `set_draft` dedup logic (a value that matches the current working is not staged as a draft). | +| 6 (`PostOpHash`) | `uint32` (only when `IncludeState: true`) | CRC32 over the PostOpValues + PostOpDrafts content. Byte-equivalent to what `GET_STATE` with the same scope + `Draft: true` would produce, so the client can prime the `PriorHash` cache. | -Each `FieldErrors` entry is itself a 3-element array: +Each `FieldErrors` entry is a 3-element array: ``` [ ns_id (uint16), field_id (uint16), error_code (uint16) ] @@ -217,18 +279,31 @@ Promote the draft layer to the working layer: - `FF_WRITE_ONLY` commands: setter fires immediately; the value is consumed and not retained. - All non-`FF_READ_ONLY`, non-`FF_WRITE_ONLY` fields' working values are persisted to per-namespace files on flash (if `RNS_USE_FS` is enabled in the firmware). -**Request payload**: optional array of namespace ids to commit. If absent, all namespaces with drafts are committed. +**Request payload**: `nil` for "commit all", or an options map: + +| Key | Value type | Meaning | +|---|---|---| +| 1 (`NamespaceFilter`) | array of `uint16` | Only commit drafts for these namespaces. | +| 5 (`IncludeState`) | `bool` | Optional. If `true`, the response includes `PostOpValues` + `PostOpHash` for the committed namespaces. See [IncludeState round-trip elimination](#includestate-round-trip-elimination). | +| 100 (`ReqCompress`) | `bool` | See [Response compression](#response-compression). | + +Example request payload (commit two namespaces, return the new state): ``` -[ 1, 101 ] # commit Reticulum and LoRa only +{ + 1: [1, 101], # NamespaceFilter — commit Reticulum + LoRa + 5: true # IncludeState +} ``` **Response payload**: a map. | Key | Value type | Meaning | |---|---|---| -| 1 (`Applied`) | `uint64` | Count of fields whose drafts were promoted. | -| 2 (`NeedsReboot`) | `bool` | `true` if any committed field has `FF_REBOOT_REQUIRED` set. Sticky from the moment of commit until a real device reboot. Distinct from `DraftHasReboot` in `SET_STATE` responses (which tracks pending drafts). | +| 1 (`Applied`) | `uint64` | Count of fields whose drafts were promoted. | +| 2 (`NeedsReboot`) | `bool` | `true` if any committed field has `FF_REBOOT_REQUIRED` set. Sticky from the moment of commit until a real device reboot. Distinct from `DraftHasReboot` in `SET_STATE` responses (which tracks pending drafts). | +| 4 (`PostOpValues`) | map (optional, only when `IncludeState: true`) | `{ ns_id: { field_id: value, ... }, ... }` — the post-commit working state for the committed namespaces. Drafts are gone (they were promoted or discarded), so there is no `PostOpDrafts`. | +| 6 (`PostOpHash`) | `uint32` (only when `IncludeState: true`) | CRC32 over `PostOpValues`. Client can prime the `GET_STATE` PriorHash cache with this hash for the same scope + `Draft: false`. | After a successful commit, the draft layer is cleared for the affected namespaces. @@ -248,7 +323,7 @@ Throw away the current draft without persisting or applying. Delete every persisted namespace file from flash and reset all working values to their declared defaults. Setters are fired with default values so the runtime reverts immediately for `FF_LIVE_APPLY` fields. `needs_reboot` is cleared. -**Request payload**: `nil`. +**Request payload**: `nil`, or an options map with `ReqCompress`. **Response payload**: a map. @@ -262,7 +337,7 @@ Integrations may register an `on_factory_reset` callback to extend the operation Ask the firmware to reboot. microReticulum itself performs no reboot — it dispatches to an `on_reboot` callback registered by the integration. If the integration registered a callback the device is expected to actually reboot (usually scheduled, so the response can be sent first); if no callback is registered the op is a successful no-op. -**Request payload**: `nil`. +**Request payload**: `nil`, or an options map with `ReqCompress`. **Response payload**: none. The response envelope is a successful ack `[op=9, seq]` with no payload element. Clients should not depend on receiving the response, since the device may be mid-reboot by the time it would have been sent. @@ -277,6 +352,41 @@ Returned in place of the normal response when the engine cannot service the requ | 1 (`ErrorCodeKey`) | `uint16` | See [Error code reference](#error-code-reference). | | 2 (`ErrorMessage`) | `str` (optional) | Free-form diagnostic. Don't show verbatim; map by code. | +## PriorHash cache short-circuit + +`GET_STATE` responses always carry a `Hash` (CRC32 over the response body content, excluding the Hash entry itself). Clients can echo that hash back on the next `GET_STATE` with the **same scope** as `PriorHash`. When it matches the server's computed hash for the current state, the server replies with a minimal `{ Unchanged: true }` body instead of re-sending the full Values/Drafts payload. + +**Cache key.** The hash is a pure function of the content the server would emit for a given `(NamespaceFilter, Draft)` combination. The client's cache key must therefore include both. A hash returned for `Draft: false, filter=[1,2]` is not comparable to a hash returned for `Draft: true, filter=[1,2]` or for `filter=[1]`. + +**When the cache hits.** For static-config namespaces the hash rarely changes between polls — only after `SET_STATE`, `COMMIT`, `DISCARD`, or `FACTORY_RESET`. Metric-heavy namespaces (fields with `FF_READ_ONLY` and a live getter, e.g. current RSSI) drift continuously and almost always miss — the response comes back with fresh Values as usual, so there's no cost besides the client's wasted PriorHash. + +**Invalidation.** After any `SET_STATE`/`COMMIT`/`DISCARD`/`FACTORY_RESET`/`REBOOT`, discard cached hashes. `SET_STATE` and `COMMIT` with `IncludeState: true` return a fresh `PostOpHash` the client can use to re-prime the cache without a follow-up `GET_STATE`. + +## IncludeState round-trip elimination + +`SET_STATE` and `COMMIT` optionally return the post-op state (working, and for `SET_STATE` also drafts) in the same response body. This saves a follow-up `GET_STATE` round-trip after a save or commit — a big win over LoRa where every request/response transaction carries fixed encryption and framing overhead in addition to the payload. + +- Set `IncludeState: true` in the request envelope to opt in. +- The response gains `PostOpValues` (working state for touched/committed namespaces), and for `SET_STATE`, `PostOpDrafts` (sparse — only namespaces/fields that hold drafts). +- The response also gains `PostOpHash` — a CRC32 over the PostOpValues + PostOpDrafts content, byte-equivalent to what a follow-up `GET_STATE` with the same scope would produce. Prime your `PriorHash` cache with it. +- **Scope of the returned state:** `SET_STATE` returns state for the namespaces referenced by the request's `State` map. `COMMIT` returns state for namespaces in the request's `NamespaceFilter` (if provided), otherwise for namespaces that actually had drafts at commit time. + +Older firmware that predates this optimization ignores `IncludeState` and simply doesn't include the PostOp keys in the response; clients should treat their absence as a fallback signal to fetch state separately. + +## Response compression + +For bulky responses (`GET_SCHEMA`, `GET_STATE` with many fields, `GET_CAPABILITIES`), the client can request that the server compress the response using **heatshrink** (LZSS-family, window=9 bits, lookahead=4 bits — matching the firmware's build flags). + +- Set `ReqCompress: true` (key 100) in the request options map. +- The server compresses only if the compressed body + wrapper is smaller than the raw payload. Otherwise it emits the uncompressed response as usual, so clients must handle both shapes. +- **Compressed responses** carry a single-key map body: + + ``` + { 101 (CompressedPayload): } + ``` + + Decode by heatshrink-decompressing the `bin` payload and then re-decoding it as MsgPack — the decoded bytes are the exact response payload the server would have emitted uncompressed. + ## Field types Wire types are intentionally a tight subset that maps cleanly to Python, JavaScript, and embedded C++. @@ -290,8 +400,9 @@ Wire types are intentionally a tight subset that maps cleanly to Python, JavaScr | 5 | `BYTES` | `bin` | `bytes` | `Uint8Array` | Opaque binary blob. Render as hex by default. | | 6 | `ENUM` | positive int on wire | `int` | `Number` | Schema carries the value list and labels; render as a dropdown. | | 7 | `BYTES_LIST`| array of `bin` | `list[bytes]` | `Array` | Variable-length list of binary blobs, often a set of fixed-size hashes. | +| 8 | `VOID` | `nil` | `None` | `null` | Argument-less command marker (write-only fields whose setter takes no value). | -`None` / `nil` values are **not** valid field values — a field is either present with a typed value or absent from a state response. +`None` / `nil` values are **not** valid field values on the wire for typed fields — a field is either present with a typed value or absent from a state response. `VOID` is a distinct type used specifically for argument-less commands (`command_void`). ## Field flags @@ -314,7 +425,7 @@ UI rendering recipes: | `FF_READ_ONLY` | Read-only display (text, gauge, status badge). Refresh via periodic `GET_STATE`. | | `FF_LIVE_APPLY | FF_SECRET` | Write-only password-style input. Never echoes the current value. | | `FF_REBOOT_REQUIRED | FF_SECRET` | Same as above, but accompanied by a "reboot required" warning after commit. | -| `FF_WRITE_ONLY` | Action button. If the field has an integer/float/string argument, present an input alongside. | +| `FF_WRITE_ONLY` | Action button. If the field has an integer/float/string argument, present an input alongside. For `VOID` fields (no argument), just the button. | ## Field schema entry @@ -380,7 +491,7 @@ schema = [ # └─ UDP ``` -`GET_STATE` responses remain **flat** — each namespace's values appear under its own id at the top level. A parent namespace can carry its own fields independently of its children. +`GET_STATE` responses remain **flat** — each namespace's values appear under its own id inside `Values` at the top level. A parent namespace can carry its own fields independently of its children. Parent ids are stable across firmware releases per the same rules as namespace ids. Once a namespace is published, its parent doesn't change. @@ -444,51 +555,76 @@ Reserved ids: A typical session for a configuration UI: 1. **Connect** to the device via your transport. Verify the link. -2. **`GET_INFO`** — confirm `SchemaVersion` is one you support and read `needs_reboot`. -3. **`GET_CAPABILITIES`** (optional) — quick sanity check that expected namespaces exist before fetching the full schema. -4. **`GET_SCHEMA`** — fetch once per session; cache by `SchemaVersion`. Build the namespace tree (parent_id) and field metadata. -5. **`GET_STATE`** — fetch current values. Render the UI. -6. As the user edits fields, send **`SET_STATE`** for each change (or batch them). Inspect `DraftHasReboot` and `FieldErrors` in each response. -7. When the user clicks "Save" / "Apply", send **`COMMIT`**. Inspect `NeedsReboot`. -8. If `NeedsReboot` is now `true`, prompt the user to reboot. The library doesn't reboot the device itself — that's the firmware/app's responsibility. -9. For commands (`FF_WRITE_ONLY`), the sequence is `SET_STATE` (argument) → `COMMIT` (triggers the setter). The next `GET_STATE` will not include the command field. -10. For metrics (`FF_READ_ONLY`), poll `GET_STATE` at whatever interval your UI needs. - -## Wire-format quick reference +2. **`GET_INFO`** — confirm `SchemaVersion` is one you support and read `needs_reboot` and `SchemaHash`. +3. **`GET_CAPABILITIES`** — fetch the namespace hierarchy with per-namespace schema hashes. Cheap; renders the top-level navigation. +4. **`GET_SCHEMA`** with `NamespaceFilter` — for each namespace whose `NsSchemaHash` isn't already in your localStorage/session cache, fetch just those. Cache per-namespace by `(ns_id, schemaHash)`. +5. **`GET_STATE`** — fetch current values (with `Draft: true` if the UI needs to show pending changes). Cache the returned `Hash` per `(scope, Draft)` combination. +6. On subsequent polls of the same scope: send the cached `Hash` back as `PriorHash`. On cache hit the response is a tiny `{ Unchanged: true }` — reuse your cached body. +7. As the user edits fields, send **`SET_STATE`** with `IncludeState: true` and the `State` map. Parse `PostOpValues` + `PostOpDrafts` from the response to refresh the panel without a follow-up `GET_STATE`. Inspect `DraftHasReboot` and `FieldErrors`. +8. When the user clicks "Save" / "Apply", send **`COMMIT`** with `IncludeState: true`. Parse `PostOpValues` to refresh the working view; inspect `NeedsReboot`. +9. If `NeedsReboot` is now `true`, prompt the user to reboot. Send **`REBOOT`** (op 9) if they agree. The library doesn't reboot the device itself — that's the firmware/app's responsibility. +10. For commands (`FF_WRITE_ONLY`), the sequence is `SET_STATE` (argument, or `nil` for `VOID` commands) → `COMMIT` (triggers the setter). The next `GET_STATE` will not include the command field. +11. For metrics (`FF_READ_ONLY`), poll `GET_STATE` at whatever interval your UI needs. `PriorHash` won't help here — live-metric hashes drift on every read. + +## Wire format quick reference ``` Envelope (always): [op_id, seq, payload] -GET_INFO request: [2, seq, nil] -GET_INFO response: [2, seq, {1: "fw_version", 2: schema_ver, 3: needs_reboot}] +GET_INFO request: [2, seq, nil | {100: compress?}] +GET_INFO response: [2, seq, {1: "fw_version", 2: schema_ver, 3: needs_reboot, 4: schema_hash}] -GET_CAPABILITIES req: [3, seq, nil] -GET_CAPABILITIES resp: [3, seq, [ns_id, ns_id, ...]] +GET_CAPABILITIES req: [3, seq, nil | {100: compress?}] +GET_CAPABILITIES resp: [3, seq, [ + {1: ns_id, 2: ns_name, 4: parent_id, 5: field_count, 6: schema_hash}, + ... + ]] -GET_SCHEMA request: [1, seq, nil] +GET_SCHEMA request: [1, seq, nil | {1: [ns_ids?], 100: compress?}] GET_SCHEMA response: [1, seq, [ [ns_id, ns_name, parent_id, [field_map, field_map, ...]], ... ]] -GET_STATE request: [4, seq, {1: [ns_ids?], 2: pending_bool?}] - (payload optional; nil OK) -GET_STATE response: [4, seq, {ns_id: {field_id: value, ...}, ...}] - -SET_STATE request: [5, seq, {ns_id: {field_id: value, ...}, ...}] -SET_STATE response: [5, seq, {1: applied, 2: draft_has_reboot, 3: [[ns,f,code], ...]?}] - -COMMIT request: [6, seq, [ns_id, ns_id, ...]?] - (array optional; nil = commit all) -COMMIT response: [6, seq, {1: applied, 2: needs_reboot}] +GET_STATE request: [4, seq, nil | {1: [ns_ids?], 2: draft?, 4: prior_hash?, 100: compress?}] +GET_STATE response (miss): + [4, seq, {1: {ns_id: {fid: value, ...}, ...}, # Values + 2: {ns_id: {fid: value, ...}, ...}?, # Drafts (optional) + 3: hash}] # Hash +GET_STATE response (hit): + [4, seq, {4: true}] # Unchanged + +SET_STATE request: [5, seq, {3: {ns_id: {fid: value, ...}, ...}, # State (required) + 5: include_state?, + 100: compress?}] +SET_STATE response: [5, seq, {1: applied, + 2: draft_has_reboot, + 3: [[ns,f,code], ...]?, # FieldErrors (optional) + 4: {ns_id: {fid: value, ...}, ...}?, # PostOpValues (optional) + 5: {ns_id: {fid: value, ...}, ...}?, # PostOpDrafts (optional) + 6: hash?}] # PostOpHash (optional) + +COMMIT request: [6, seq, nil | {1: [ns_ids?], 5: include_state?, 100: compress?}] +COMMIT response: [6, seq, {1: applied, + 2: needs_reboot, + 4: {ns_id: {fid: value, ...}, ...}?, # PostOpValues (optional) + 6: hash?}] # PostOpHash (optional) DISCARD request: [7, seq, [ns_id, ns_id, ...]?] DISCARD response: [7, seq, {1: applied}] -FACTORY_RESET request: [8, seq, nil] +FACTORY_RESET request: [8, seq, nil | {100: compress?}] FACTORY_RESET response: [8, seq, {2: needs_reboot_false}] +REBOOT request: [9, seq, nil | {100: compress?}] +REBOOT response: [9, seq] # no payload — device may be mid-reboot + ERROR response: [101, seq, {1: error_code, 2: "message"?}] + +COMPRESSED response: the payload is replaced by a single-entry map + {101 (CompressedPayload): } whose value + is a heatshrink-encoded MsgPack blob that + decompresses to the raw payload above. ``` --- diff --git a/docs/provisioning_integration_guide.md b/docs/provisioning_integration_guide.md index b8b5f480..4d3af6a7 100644 --- a/docs/provisioning_integration_guide.md +++ b/docs/provisioning_integration_guide.md @@ -355,8 +355,8 @@ Mechanics: - Each `.register_namespace()` call pushes onto a per-`Provisioner` scope stack and uses the current scope top as the new namespace's parent. - Each `.end()` pops one level. Forgetting `.end()` is caught at `Provisioner::begin()` — the scope stack must be empty by the time `begin()` runs (or it gets cleared with a warning). - The Registry remains **flat** internally — every namespace is a top-level entry with an optional `parent_id`. Lookups stay O(1) by id. -- On disk: `Interfaces.msgpack`, `Interfaces.LoRa.msgpack`, `Interfaces.UDP.msgpack`. Three files at the top level of the storage directory. -- On the wire: `GET_STATE` keeps each namespace at the top level keyed by its own id. The tree shape is conveyed only through the schema's `parent_id`. Clients reconstruct the tree client-side. +- On disk: `ns.msgpack` — one file per namespace, keyed by numeric id. The tree shape is not reflected in filenames. +- On the wire: `GET_STATE` returns values in a flat `{ns_id: {field_id: value, ...}, ...}` map (nested inside the response body's `Values` key). The tree shape is conveyed only through the schema's `parent_id`. Clients reconstruct the tree client-side. Cycle detection runs at registration time — declaring a parent that walks up to the new namespace itself is refused. @@ -724,7 +724,8 @@ public: Registry& registry(); Storage* storage(); - static constexpr uint16_t SCHEMA_VERSION = 1; + static constexpr uint16_t SCHEMA_VERSION = 2; + uint32_t schema_hash() const; }; ``` @@ -766,7 +767,7 @@ using CommitCallback = std::function; ### `Value` (tagged variant) ```cpp -enum class Type : uint8_t { None=0, Bool=1, Int=2, Float=3, String=4, Bytes=5, Enum=6, BytesList=7 }; +enum class Type : uint8_t { None=0, Bool=1, Int=2, Float=3, String=4, Bytes=5, Enum=6, BytesList=7, Void=8 }; class Value { public: diff --git a/src/microReticulum/Provisioning/Namespace.h b/src/microReticulum/Provisioning/Namespace.h index c82a0e96..129ca598 100644 --- a/src/microReticulum/Provisioning/Namespace.h +++ b/src/microReticulum/Provisioning/Namespace.h @@ -71,6 +71,9 @@ namespace RNS { namespace Provisioning { bool draft(fid_t field_id, Value& out) const; bool has_draft(fid_t field_id) const; + // True iff at least one draft entry exists in this namespace. + bool has_any_draft() const { return !_draft.empty(); } + // Write to draft after validating against the field's constraint. // Returns false if the field is unknown, read-only, or invalid. bool set_draft(fid_t field_id, const Value& v); @@ -110,6 +113,13 @@ namespace RNS { namespace Provisioning { bool has_on_commit() const { return (bool)_on_commit; } const CommitCallback& on_commit_callback() const { return _on_commit; } + // CRC32 of this namespace's schema entry as it would appear in the + // GetSchema payload. Computed by Provisioner::begin() after all fields + // are registered and surfaced in GetCapabilities so clients can cache + // each namespace's schema independently. + uint32_t schema_hash() const { return _schema_hash; } + void schema_hash(uint32_t h) { _schema_hash = h; } + private: nid_t _id; fstring_t _name; @@ -120,6 +130,7 @@ namespace RNS { namespace Provisioning { std::unordered_map _working; std::unordered_map _draft; CommitCallback _on_commit; + uint32_t _schema_hash = 0; bool _dirty_for_persist = false; }; diff --git a/src/microReticulum/Provisioning/Ops.h b/src/microReticulum/Provisioning/Ops.h index 3c425fc0..02b0e8b1 100644 --- a/src/microReticulum/Provisioning/Ops.h +++ b/src/microReticulum/Provisioning/Ops.h @@ -69,11 +69,27 @@ namespace RNS { namespace Provisioning { constexpr uint16_t NeedsReboot = 2; constexpr uint16_t DraftHasReboot = 2; // alias for SetState constexpr uint16_t FieldErrors = 3; + // Post-op state returned in Commit and SetState responses when + // IncludeState was requested. Slots 4-6 avoid collision with + // Applied(1) / NeedsReboot(2) / FieldErrors(3) in the enclosing + // response body. Commit only emits Values+Hash (drafts are cleared + // by commit); SetState emits Values+Drafts+Hash (drafts persist). + constexpr uint16_t PostOpValues = 4; + constexpr uint16_t PostOpDrafts = 5; + constexpr uint16_t PostOpHash = 6; - // GetState / SetState request flags + // GetState / SetState / Commit request flags constexpr uint16_t NamespaceFilter = 1; - constexpr uint16_t Pending = 2; + constexpr uint16_t Draft = 2; // GetState request: true = also include drafts alongside working constexpr uint16_t State = 3; // the {ns: {field: value}} map + constexpr uint16_t PriorHash = 4; // GetState request: client-supplied CRC32 for cache short-circuit + constexpr uint16_t IncludeState = 5; // Commit request: return post-commit Values in the response + + // GetState response body + constexpr uint16_t Values = 1; // {ns_id: {field_id: value}} map (working values) + constexpr uint16_t Drafts = 2; // sparse {ns_id: {field_id: value}} — only ns/fields with drafts + constexpr uint16_t Hash = 3; // CRC32 over the response body; echoed back as PriorHash + constexpr uint16_t Unchanged = 4; // present iff the client's PriorHash matched // GetInfo constexpr uint16_t FirmwareVersion = 1; @@ -90,6 +106,8 @@ namespace RNS { namespace Provisioning { constexpr uint16_t NsName = 2; constexpr uint16_t NsFields = 3; constexpr uint16_t NsParent = 4; // optional parent ns id (0 = root) + constexpr uint16_t NsFieldCount = 5; // GetCapabilities map: number of fields + constexpr uint16_t NsSchemaHash = 6; // GetCapabilities map: per-namespace CRC32 constexpr uint16_t FieldId = 1; constexpr uint16_t FieldName = 2; constexpr uint16_t FieldType = 3; diff --git a/src/microReticulum/Provisioning/Provisioning.cpp b/src/microReticulum/Provisioning/Provisioning.cpp index 25818f09..abff63d2 100644 --- a/src/microReticulum/Provisioning/Provisioning.cpp +++ b/src/microReticulum/Provisioning/Provisioning.cpp @@ -27,11 +27,12 @@ namespace RNS { namespace Provisioning { // Defined in BuiltinNamespaces.cpp. void register_builtin_namespaces(Provisioner& p); - // Forward declaration — definition lives next to op_get_schema below so + // Forward declarations — definitions live next to op_get_schema below so // the schema serialization logic stays grouped. Called from begin() to - // compute the boot-time schema hash and from op_get_schema for the wire - // path so the hash matches the bytes clients actually receive. + // compute the boot-time schema hashes and from op_get_schema for the + // wire path so the hashes match the bytes clients actually receive. static void pack_schema_payload(MsgPack::Packer& p, const Registry& registry); + static void pack_namespace_schema(MsgPack::Packer& p, const Namespace& ns); // --------------------------------------------------------------------- // Singleton @@ -73,7 +74,14 @@ namespace RNS { namespace Provisioning { // once here after registration is complete so GetInfo can report // it as a cache key — clients keying their local schema cache by // this hash skip a full GetSchema fetch when the device's schema - // has not changed. + // has not changed. Per-namespace hashes are also cached so lazy + // per-namespace schema fetches (via GetCapabilities + filtered + // GetSchema) can validate individual namespaces by hash. + for (const auto& ns_ptr : _registry.namespaces()) { + MsgPack::Packer nhp; + pack_namespace_schema(nhp, *ns_ptr); + ns_ptr->schema_hash(Utilities::Crc::crc32(0, (const uint8_t*)nhp.data(), nhp.size())); + } { MsgPack::Packer hp; pack_schema_payload(hp, _registry); @@ -211,11 +219,7 @@ namespace RNS { namespace Provisioning { // commit". The callback may revert (clear_draft) or amend // (set_draft) drafts, so we re-collect the id list after it // returns to honour those edits. - bool has_any_draft = false; - for (const Field& f : ns.fields()) { - if (ns.has_draft(f.id)) { has_any_draft = true; break; } - } - if (has_any_draft && ns.has_on_commit()) { + if (ns.has_any_draft() && ns.has_on_commit()) { try { ns.on_commit_callback()(ns); } catch (const std::exception& e) { ERRORF("Provisioning::commit: on_commit callback for namespace %u threw: %s", @@ -469,12 +473,24 @@ namespace RNS { namespace Provisioning { } Bytes Provisioner::op_get_capabilities(seq_t seq, void* unpacker_v) { + // Returns an array describing every registered namespace — the "map" + // clients use at connect time to discover the namespace hierarchy and + // decide which schemas need to be fetched. Each entry is a small map + // keyed by the Ns* constants in Ops.h. The per-namespace schema hash + // lets clients cache each namespace's schema independently and skip + // re-fetching when only some namespaces have changed. const bool compress = parse_compress_only(unpacker_v); return pack_response((opid_t)Op::GetCapabilities, seq, compress, [&](MsgPack::Packer& p) { const auto& nss = _registry.namespaces(); p.serialize(MsgPack::arr_size_t(nss.size())); for (const auto& ns_ptr : nss) { - p.serialize((nid_t)ns_ptr->id()); + const Namespace& ns = *ns_ptr; + p.serialize(MsgPack::map_size_t(5)); + p.serialize((uint16_t)Key::NsId); p.serialize((nid_t)ns.id()); + p.serialize((uint16_t)Key::NsName); p.serialize(ns.name().c_str()); + p.serialize((uint16_t)Key::NsParent); p.serialize((nid_t)ns.parent_id()); + p.serialize((uint16_t)Key::NsFieldCount); p.serialize((uint64_t)ns.fields().size()); + p.serialize((uint16_t)Key::NsSchemaHash); p.serialize((uint32_t)ns.schema_hash()); } }); } @@ -525,72 +541,120 @@ namespace RNS { namespace Provisioning { return n; } - // Packs the GetSchema payload (just the namespaces array) onto `p`. - // Shared by op_get_schema() and the boot-time CRC32 hashing path so - // the hash is computed over the exact bytes clients receive. + // Packs one namespace's schema entry: [id, name, parent_id, [field-maps]]. + // parent_id of 0 means root (no parent). Schema v2 layout — v1 clients + // reading the first three elements still parse the rest correctly. Also + // used by begin() to CRC32 each namespace independently so clients can + // cache per-namespace schemas. + static void pack_namespace_schema(MsgPack::Packer& p, const Namespace& ns) { + p.serialize(MsgPack::arr_size_t(4)); + p.serialize((nid_t)ns.id()); + p.serialize(ns.name().c_str()); + p.serialize((nid_t)ns.parent_id()); + const auto& fields = ns.fields(); + p.serialize(MsgPack::arr_size_t(fields.size())); + for (const Field& f : fields) { + p.serialize(MsgPack::map_size_t(schema_field_entries(f))); + p.serialize((uint16_t)Key::FieldId); p.serialize((fid_t)f.id); + p.serialize((uint16_t)Key::FieldName); p.serialize(f.name.c_str()); + p.serialize((uint16_t)Key::FieldType); p.serialize((uint8_t)f.type); + p.serialize((uint16_t)Key::FieldFlags); p.serialize((fflags_t)f.flags); + p.serialize((uint16_t)Key::FieldDefault); pack_field_default(p, f); + if (f.type == Type::Int && f.constraint.has_range) { + p.serialize((uint16_t)Key::FieldMinI); p.serialize((fint_t)f.constraint.imin); + p.serialize((uint16_t)Key::FieldMaxI); p.serialize((fint_t)f.constraint.imax); + } + if (f.type == Type::Float && f.constraint.has_range) { + p.serialize((uint16_t)Key::FieldMinF); p.serialize((double)f.constraint.fmin); + p.serialize((uint16_t)Key::FieldMaxF); p.serialize((double)f.constraint.fmax); + } + if ((f.type == Type::String || f.type == Type::Bytes) && f.constraint.max_len > 0) { + p.serialize((uint16_t)Key::FieldMaxLen); p.serialize((uint64_t)f.constraint.max_len); + } + if (f.type == Type::Enum) { + if (!f.constraint.enum_values.empty()) { + p.serialize((uint16_t)Key::FieldEnumValues); + p.serialize(MsgPack::arr_size_t(f.constraint.enum_values.size())); + for (fenum_t v : f.constraint.enum_values) p.serialize(v); + } + if (!f.constraint.enum_labels.empty()) { + p.serialize((uint16_t)Key::FieldEnumLabels); + p.serialize(MsgPack::arr_size_t(f.constraint.enum_labels.size())); + for (const auto& s : f.constraint.enum_labels) p.serialize(s.c_str()); + } + } + if (f.type == Type::BytesList) { + if (f.constraint.element_size > 0) { + p.serialize((uint16_t)Key::FieldElementSize); + p.serialize((uint64_t)f.constraint.element_size); + } + if (f.constraint.max_count > 0) { + p.serialize((uint16_t)Key::FieldMaxCount); + p.serialize((uint64_t)f.constraint.max_count); + } + } + } + } + + // Packs the full GetSchema payload (namespaces array) onto `p`. static void pack_schema_payload(MsgPack::Packer& p, const Registry& registry) { const auto& nss = registry.namespaces(); p.serialize(MsgPack::arr_size_t(nss.size())); for (const auto& ns_ptr : nss) { - const Namespace& ns = *ns_ptr; - // Each namespace is [id, name, parent_id_or_zero, [field-maps]]. - // parent_id of 0 means root (no parent). Schema v2 layout — - // v1 clients reading the first three elements still parse - // the rest of the response correctly. - p.serialize(MsgPack::arr_size_t(4)); - p.serialize((nid_t)ns.id()); - p.serialize(ns.name().c_str()); - p.serialize((nid_t)ns.parent_id()); - const auto& fields = ns.fields(); - p.serialize(MsgPack::arr_size_t(fields.size())); - for (const Field& f : fields) { - p.serialize(MsgPack::map_size_t(schema_field_entries(f))); - p.serialize((uint16_t)Key::FieldId); p.serialize((fid_t)f.id); - p.serialize((uint16_t)Key::FieldName); p.serialize(f.name.c_str()); - p.serialize((uint16_t)Key::FieldType); p.serialize((uint8_t)f.type); - p.serialize((uint16_t)Key::FieldFlags); p.serialize((fflags_t)f.flags); - p.serialize((uint16_t)Key::FieldDefault); pack_field_default(p, f); - if (f.type == Type::Int && f.constraint.has_range) { - p.serialize((uint16_t)Key::FieldMinI); p.serialize((fint_t)f.constraint.imin); - p.serialize((uint16_t)Key::FieldMaxI); p.serialize((fint_t)f.constraint.imax); - } - if (f.type == Type::Float && f.constraint.has_range) { - p.serialize((uint16_t)Key::FieldMinF); p.serialize((double)f.constraint.fmin); - p.serialize((uint16_t)Key::FieldMaxF); p.serialize((double)f.constraint.fmax); - } - if ((f.type == Type::String || f.type == Type::Bytes) && f.constraint.max_len > 0) { - p.serialize((uint16_t)Key::FieldMaxLen); p.serialize((uint64_t)f.constraint.max_len); - } - if (f.type == Type::Enum) { - if (!f.constraint.enum_values.empty()) { - p.serialize((uint16_t)Key::FieldEnumValues); - p.serialize(MsgPack::arr_size_t(f.constraint.enum_values.size())); - for (fenum_t v : f.constraint.enum_values) p.serialize(v); - } - if (!f.constraint.enum_labels.empty()) { - p.serialize((uint16_t)Key::FieldEnumLabels); - p.serialize(MsgPack::arr_size_t(f.constraint.enum_labels.size())); - for (const auto& s : f.constraint.enum_labels) p.serialize(s.c_str()); + pack_namespace_schema(p, *ns_ptr); + } + } + + Bytes Provisioner::op_get_schema(seq_t seq, void* unpacker_v) { + // Optional request payload: { NamespaceFilter: [ns_ids], ReqCompress: bool }. + // Nil payload or missing filter → full schema (unchanged behavior). + // With a filter → only the listed namespaces are packed, preserving + // registry order. Unknown ids in the filter are silently ignored; + // callers can detect this by cross-checking against GetCapabilities. + MsgPack::Unpacker* up = (MsgPack::Unpacker*)unpacker_v; + std::unordered_set ns_filter; + bool has_filter = false; + bool compress = false; + if (up && up->isMap()) { + const size_t n = up->unpackMapSize(); + for (size_t i = 0; i < n; ++i) { + nid_t key; + if (!read_uint_key(*up, key)) { skip_value(*up); continue; } + if (key == Key::NamespaceFilter) { + if (up->isArray()) { + const size_t m = up->unpackArraySize(); + for (size_t j = 0; j < m; ++j) { + if (up->isUInt() || up->isInt()) { + fint_t v; up->deserialize(v); + ns_filter.insert((nid_t)v); + } + else skip_value(*up); + } + has_filter = true; } + else skip_value(*up); } - if (f.type == Type::BytesList) { - if (f.constraint.element_size > 0) { - p.serialize((uint16_t)Key::FieldElementSize); - p.serialize((uint64_t)f.constraint.element_size); - } - if (f.constraint.max_count > 0) { - p.serialize((uint16_t)Key::FieldMaxCount); - p.serialize((uint64_t)f.constraint.max_count); - } + else if (key == Key::ReqCompress) { + if (up->isBool()) up->deserialize(compress); + else skip_value(*up); } + else skip_value(*up); } } - } + else if (up) skip_value(*up); - Bytes Provisioner::op_get_schema(seq_t seq, void* unpacker_v) { - const bool compress = parse_compress_only(unpacker_v); return pack_response((opid_t)Op::GetSchema, seq, compress, [&](MsgPack::Packer& p) { - pack_schema_payload(p, _registry); + if (!has_filter) { + pack_schema_payload(p, _registry); + return; + } + // Filtered: emit only the requested namespaces, in registry order. + std::vector selected; + for (const auto& ns_ptr : _registry.namespaces()) { + if (ns_filter.count(ns_ptr->id()) != 0) selected.push_back(ns_ptr.get()); + } + p.serialize(MsgPack::arr_size_t(selected.size())); + for (const Namespace* ns : selected) pack_namespace_schema(p, *ns); }); } @@ -599,13 +663,90 @@ namespace RNS { namespace Provisioning { Codec::pack_value(p, v); } + // FF_REBOOT_REQUIRED reads return the working (committed-but-pending-reboot) + // value, not the live runtime via effective()'s getter — mirrors + // Codec::persist_value. Operators are entitled to see the canonical committed + // value across reconnects and reader identities; the live-runtime value of a + // reboot-gated field would diverge between commit and reboot and disagree + // across clients. + static Value base_state_value(const Namespace& n, const Field& fl) { + return fl.has_flag(FF_REBOOT_REQUIRED) ? n.working(fl.id) : n.effective(fl.id); + } + + // Packs the `Values` sub-map { ns_id: { fid: value, ... }, ... } for the + // given namespaces. FF_SECRET and FF_WRITE_ONLY fields are excluded, none- + // typed values are dropped. Shared by op_get_state, op_commit, and + // op_set_state so all emit byte-identical Values content for the same + // underlying state. + static void pack_ns_values(MsgPack::Packer& p, const std::vector& ns_list) { + p.serialize(MsgPack::map_size_t(ns_list.size())); + for (const Namespace* ns : ns_list) { + p.serialize((nid_t)ns->id()); + size_t fld_entries = 0; + for (const Field& f : ns->fields()) { + if (f.has_flag(FF_SECRET)) continue; + if (f.has_flag(FF_WRITE_ONLY)) continue; + Value v = base_state_value(*ns, f); + if (!v.is_none()) ++fld_entries; + } + p.serialize(MsgPack::map_size_t(fld_entries)); + for (const Field& f : ns->fields()) { + if (f.has_flag(FF_SECRET)) continue; + if (f.has_flag(FF_WRITE_ONLY)) continue; + Value v = base_state_value(*ns, f); + if (v.is_none()) continue; + p.serialize((fid_t)f.id); + pack_state_value(p, f.type, v); + } + } + } + + // Packs a sparse Drafts sub-map { ns_id: { fid: draft_value, ... }, ... } + // for the given namespaces (call with only namespaces that actually have + // drafts). Shared by op_get_state and op_set_state so both emit + // byte-identical Drafts content — lets the client cache-prime a GetState + // response with a hash returned by SetState. + static void pack_ns_drafts(MsgPack::Packer& p, const std::vector& draft_ns_list) { + p.serialize(MsgPack::map_size_t(draft_ns_list.size())); + for (const Namespace* ns : draft_ns_list) { + p.serialize((nid_t)ns->id()); + size_t draft_entries = 0; + for (const Field& f : ns->fields()) { + if (ns->has_draft(f.id)) ++draft_entries; + } + p.serialize(MsgPack::map_size_t(draft_entries)); + for (const Field& f : ns->fields()) { + Value v; + if (!ns->draft(f.id, v)) continue; + p.serialize((fid_t)f.id); + pack_state_value(p, f.type, v); + } + } + } + Bytes Provisioner::op_get_state(seq_t seq, void* unpacker_v) { + // Request payload (all optional): + // { NamespaceFilter: [ns_ids], Draft: bool, PriorHash: uint32, ReqCompress: bool } + // + // Response body: + // { Values: {ns: {fid: value, ...}, ...}, Drafts?: {ns: {fid: value, ...}, ...}, Hash: uint32 } + // + // Draft=true asks the server to include drafts alongside working in the + // same response. The Drafts sub-map is sparse — only namespaces that + // have pending drafts, and only fields with drafts. Omitted entirely + // when no in-scope namespace has drafts (Draft=false or drafts empty). + // + // PriorHash short-circuit: if the client-supplied hash matches the CRC32 + // the server would compute over the response body (excluding the Hash + // entry itself), the body collapses to { Unchanged: true } and the + // client reuses its cache. MsgPack::Unpacker* up = (MsgPack::Unpacker*)unpacker_v; std::unordered_set ns_filter; bool has_filter = false; - bool pending = false; + bool include_drafts = false; + bool has_prior_hash = false; + uint32_t prior_hash = 0; bool compress = false; - // Optional payload map: {1: [ns_filter], 2: pending, 100: compress} if (up && up->isMap()) { const size_t n = up->unpackMapSize(); for (size_t i = 0; i < n; ++i) { @@ -625,8 +766,16 @@ namespace RNS { namespace Provisioning { } else skip_value(*up); } - else if (key == Key::Pending) { - if (up->isBool()) up->deserialize(pending); + else if (key == Key::Draft) { + if (up->isBool()) up->deserialize(include_drafts); + else skip_value(*up); + } + else if (key == Key::PriorHash) { + if (up->isUInt() || up->isInt()) { + fint_t v = 0; up->deserialize(v); + prior_hash = (uint32_t)v; + has_prior_hash = true; + } else skip_value(*up); } else if (key == Key::ReqCompress) { @@ -638,109 +787,190 @@ namespace RNS { namespace Provisioning { } else if (up) skip_value(*up); - return pack_response((opid_t)Op::GetState, seq, compress, [&](MsgPack::Packer& p) { - std::vector ns_list; - for (const auto& ns_ptr : _registry.namespaces()) { - if (has_filter && ns_filter.count(ns_ptr->id()) == 0) continue; - ns_list.push_back(ns_ptr.get()); - } - p.serialize(MsgPack::map_size_t(ns_list.size())); + // In-scope namespaces (registry order, respecting NamespaceFilter). + std::vector ns_list; + for (const auto& ns_ptr : _registry.namespaces()) { + if (has_filter && ns_filter.count(ns_ptr->id()) == 0) continue; + ns_list.push_back(ns_ptr.get()); + } + + // Sub-list of in-scope namespaces that have pending drafts. Only + // consulted when the client asked for drafts alongside working. + std::vector draft_ns_list; + if (include_drafts) { for (const Namespace* ns : ns_list) { - p.serialize((nid_t)ns->id()); - // For FF_REBOOT_REQUIRED fields the read returns the working - // (committed-but-pending-reboot) value, not the live runtime - // via effective()'s getter — see Codec::persist_value for the - // matching persistence rule. Operators are entitled to see - // the canonical committed value across reconnects, transports, - // and reader identities; the live-runtime value of a reboot- - // gated field would diverge between commit and reboot and - // would disagree across clients reading the same device. - auto base_value = [](const Namespace& n, const Field& fl) { - return fl.has_flag(FF_REBOOT_REQUIRED) ? n.working(fl.id) : n.effective(fl.id); - }; - // Count entries first (skip SECRET fields). - size_t entries = 0; - for (const Field& f : ns->fields()) { - if (f.has_flag(FF_SECRET)) continue; - if (f.has_flag(FF_WRITE_ONLY)) continue; - Value v = base_value(*ns, f); - if (pending) { - Value d; - if (ns->draft(f.id, d)) v = d; - } - if (!v.is_none()) ++entries; - } - p.serialize(MsgPack::map_size_t(entries)); - for (const Field& f : ns->fields()) { - if (f.has_flag(FF_SECRET)) continue; - Value v = base_value(*ns, f); - if (pending) { - Value d; - if (ns->draft(f.id, d)) v = d; - } - if (v.is_none()) continue; - p.serialize((fid_t)f.id); - pack_state_value(p, f.type, v); - } + if (ns->has_any_draft()) draft_ns_list.push_back(ns); + } + } + + // Emit the content map (Values + optional Drafts) into a packer, + // with_hash controlling whether a Hash entry is appended. Called + // twice: once to compute the CRC over the hash-less form, and (on + // cache miss) once inside pack_response with the actual hash. + auto pack_content = [&](MsgPack::Packer& p, bool with_hash, uint32_t hash) { + const bool emit_drafts = include_drafts && !draft_ns_list.empty(); + size_t entries = 1; // Values always + if (emit_drafts) ++entries; + if (with_hash) ++entries; + p.serialize(MsgPack::map_size_t(entries)); + + p.serialize((uint16_t)Key::Values); + pack_ns_values(p, ns_list); + + if (emit_drafts) { + p.serialize((uint16_t)Key::Drafts); + pack_ns_drafts(p, draft_ns_list); + } + + if (with_hash) { + p.serialize((uint16_t)Key::Hash); + p.serialize((uint32_t)hash); } + }; + + // Hash pass: pack the content without the Hash entry and CRC the + // resulting bytes. Deterministic given the same in-memory state. + uint32_t body_hash = 0; + { + MsgPack::Packer scratch; + pack_content(scratch, false, 0); + body_hash = Utilities::Crc::crc32(0, (const uint8_t*)scratch.data(), scratch.size()); + } + + return pack_response((opid_t)Op::GetState, seq, compress, [&](MsgPack::Packer& p) { + if (has_prior_hash && prior_hash == body_hash) { + p.serialize(MsgPack::map_size_t(1)); + p.serialize((uint16_t)Key::Unchanged); + p.serialize((bool)true); + return; + } + pack_content(p, true, body_hash); }); } Bytes Provisioner::op_set_state(seq_t seq, void* unpacker_v) { + // Request envelope: + // { State: { ns_id: { fid: value, ... }, ... }, // required + // IncludeState: bool, // optional + // ReqCompress: bool } // optional + // + // Response body: + // { Applied, DraftHasReboot, FieldErrors?, + // PostOpValues?, PostOpDrafts?, PostOpHash? } + // PostOpValues/PostOpDrafts/PostOpHash appear iff IncludeState was true. + // PostOpValues is the working state for the namespaces referenced by the + // request (working is unchanged by SetState; still emitted so the client + // can refresh its cache without a follow-up GetState). PostOpDrafts is + // the sparse per-namespace draft map after the writes were applied, + // including whatever set_draft() dedup logic produced. MsgPack::Unpacker* up = (MsgPack::Unpacker*)unpacker_v; if (!up || !up->isMap()) { return encode_error((opid_t)Op::SetState, seq, ErrorCode::MalformedRequest, "expected map payload"); } - const size_t n_ns = up->unpackMapSize(); size_t applied = 0; struct Err { nid_t ns_id; fid_t field_id; ferror_t code; }; std::vector errors; - for (size_t i = 0; i < n_ns; ++i) { - if (!(up->isUInt() || up->isInt())) { skip_value(*up); skip_value(*up); continue; } - fint_t k1 = 0; up->deserialize(k1); - const nid_t ns_id = (nid_t)k1; - Namespace* ns = _registry.find(ns_id); - if (!up->isMap()) { - // Inner must be a map of {field_id: value} + bool include_state = false; + bool compress = false; + bool saw_state = false; + // Namespaces referenced by the State map, in first-seen order, deduped. + // Used to scope the PostOpValues/PostOpDrafts response when IncludeState + // is true. Kept as a Namespace* list so we can iterate fields without a + // second registry lookup. + std::vector touched_ns; + std::unordered_set touched_seen; + + const size_t n_top = up->unpackMapSize(); + for (size_t t = 0; t < n_top; ++t) { + nid_t top_key; + if (!read_uint_key(*up, top_key)) { skip_value(*up); continue; } + if (top_key == Key::IncludeState) { + if (up->isBool()) up->deserialize(include_state); + else skip_value(*up); + continue; + } + if (top_key == Key::ReqCompress) { + if (up->isBool()) up->deserialize(compress); + else skip_value(*up); + continue; + } + if (top_key != Key::State) { + // Unknown envelope key — skip its value. skip_value(*up); - if (!ns) errors.push_back({ns_id, 0, (ferror_t)ErrorCode::UnknownNamespace}); - else errors.push_back({ns_id, 0, (ferror_t)ErrorCode::MalformedRequest}); continue; } - const size_t n_fields = up->unpackMapSize(); - for (size_t j = 0; j < n_fields; ++j) { + // State: process the inner { ns_id: { fid: value } } map. + saw_state = true; + if (!up->isMap()) { skip_value(*up); continue; } + const size_t n_ns = up->unpackMapSize(); + for (size_t i = 0; i < n_ns; ++i) { if (!(up->isUInt() || up->isInt())) { skip_value(*up); skip_value(*up); continue; } - fint_t k2 = 0; up->deserialize(k2); - const fid_t fid = (fid_t)k2; - if (!ns) { + fint_t k1 = 0; up->deserialize(k1); + const nid_t ns_id = (nid_t)k1; + Namespace* ns = _registry.find(ns_id); + if (ns && touched_seen.insert(ns_id).second) touched_ns.push_back(ns); + if (!up->isMap()) { skip_value(*up); - errors.push_back({ns_id, fid, (ferror_t)ErrorCode::UnknownNamespace}); - continue; - } - const Field* f = ns->find_field(fid); - if (!f) { - skip_value(*up); - errors.push_back({ns_id, fid, (ferror_t)ErrorCode::UnknownField}); - continue; - } - Value v; - if (!Codec::unpack_value(*up, f->type, v)) { - errors.push_back({ns_id, fid, (ferror_t)ErrorCode::InvalidValue}); - continue; - } - if (f->has_flag(FF_READ_ONLY)) { - errors.push_back({ns_id, fid, (ferror_t)ErrorCode::ReadOnly}); + if (!ns) errors.push_back({ns_id, 0, (ferror_t)ErrorCode::UnknownNamespace}); + else errors.push_back({ns_id, 0, (ferror_t)ErrorCode::MalformedRequest}); continue; } - if (!ns->set_draft(fid, v)) { - errors.push_back({ns_id, fid, (ferror_t)ErrorCode::ConstraintViolation}); - continue; + const size_t n_fields = up->unpackMapSize(); + for (size_t j = 0; j < n_fields; ++j) { + if (!(up->isUInt() || up->isInt())) { skip_value(*up); skip_value(*up); continue; } + fint_t k2 = 0; up->deserialize(k2); + const fid_t fid = (fid_t)k2; + if (!ns) { + skip_value(*up); + errors.push_back({ns_id, fid, (ferror_t)ErrorCode::UnknownNamespace}); + continue; + } + const Field* f = ns->find_field(fid); + if (!f) { + skip_value(*up); + errors.push_back({ns_id, fid, (ferror_t)ErrorCode::UnknownField}); + continue; + } + Value v; + if (!Codec::unpack_value(*up, f->type, v)) { + errors.push_back({ns_id, fid, (ferror_t)ErrorCode::InvalidValue}); + continue; + } + if (f->has_flag(FF_READ_ONLY)) { + errors.push_back({ns_id, fid, (ferror_t)ErrorCode::ReadOnly}); + continue; + } + if (!ns->set_draft(fid, v)) { + errors.push_back({ns_id, fid, (ferror_t)ErrorCode::ConstraintViolation}); + continue; + } + ++applied; } - ++applied; } } - return pack_response((opid_t)Op::SetState, seq, [&](MsgPack::Packer& p) { - p.serialize(MsgPack::map_size_t(errors.empty() ? 2 : 3)); + if (!saw_state) { + return encode_error((opid_t)Op::SetState, seq, ErrorCode::MalformedRequest, "missing State key"); + } + + // Determine which of the touched namespaces have drafts *after* the + // writes were applied — sparse Drafts map only lists non-empty ones. + std::vector touched_with_drafts; + if (include_state) { + for (const Namespace* ns : touched_ns) { + if (ns->has_any_draft()) touched_with_drafts.push_back(ns); + } + } + + return pack_response((opid_t)Op::SetState, seq, compress, [&](MsgPack::Packer& p) { + // Base entries: Applied + DraftHasReboot. Plus optional + // FieldErrors, plus optional PostOpValues/PostOpDrafts/PostOpHash. + size_t entries = 2; + if (!errors.empty()) ++entries; + if (include_state) { + entries += 2; // PostOpValues + PostOpHash + if (!touched_with_drafts.empty()) ++entries; + } + p.serialize(MsgPack::map_size_t(entries)); p.serialize((uint16_t)Key::Applied); p.serialize((uint64_t)applied); p.serialize((uint16_t)Key::DraftHasReboot); p.serialize((bool)draft_has_reboot()); if (!errors.empty()) { @@ -753,20 +983,74 @@ namespace RNS { namespace Provisioning { p.serialize((ferror_t)e.code); } } + if (include_state) { + // Build the PostOp content into a scratch, CRC32 it, then + // re-emit inline alongside the hash. Hash covers just the + // Values+Drafts content (excludes Applied/DraftHasReboot + // framing) so it matches what a follow-up GetState with the + // same scope + withDrafts:true would compute. + MsgPack::Packer scratch; + pack_ns_values(scratch, touched_ns); + if (!touched_with_drafts.empty()) { + pack_ns_drafts(scratch, touched_with_drafts); + } + const uint32_t hash = Utilities::Crc::crc32(0, + (const uint8_t*)scratch.data(), scratch.size()); + + p.serialize((uint16_t)Key::PostOpValues); + pack_ns_values(p, touched_ns); + if (!touched_with_drafts.empty()) { + p.serialize((uint16_t)Key::PostOpDrafts); + pack_ns_drafts(p, touched_with_drafts); + } + p.serialize((uint16_t)Key::PostOpHash); + p.serialize((uint32_t)hash); + } }); } Bytes Provisioner::op_commit(seq_t seq, void* unpacker_v) { + // Request payload (all optional): + // { NamespaceFilter: [ns_ids], IncludeState: bool, ReqCompress: bool } + // Nil payload = commit all namespaces, IncludeState=false. + // + // Response body: + // { Applied: N, NeedsReboot: bool, Values?: {ns: {fid: value}, ...}, Hash?: uint32 } + // Values and Hash appear iff IncludeState was true. The Values map is + // the post-commit working state for the committed namespaces (same + // content the client would see from a follow-up GetState), letting + // callers refresh the UI without an extra round-trip. MsgPack::Unpacker* up = (MsgPack::Unpacker*)unpacker_v; std::vector filter; bool has_filter = false; - if (up && up->isArray()) { - const size_t n = up->unpackArraySize(); - has_filter = true; + bool include_state = false; + bool compress = false; + if (up && up->isMap()) { + const size_t n = up->unpackMapSize(); for (size_t i = 0; i < n; ++i) { - if (up->isUInt() || up->isInt()) { - fint_t v; up->deserialize(v); - filter.push_back((nid_t)v); + nid_t key; + if (!read_uint_key(*up, key)) { skip_value(*up); continue; } + if (key == Key::NamespaceFilter) { + if (up->isArray()) { + const size_t m = up->unpackArraySize(); + for (size_t j = 0; j < m; ++j) { + if (up->isUInt() || up->isInt()) { + fint_t v; up->deserialize(v); + filter.push_back((nid_t)v); + } + else skip_value(*up); + } + has_filter = true; + } + else skip_value(*up); + } + else if (key == Key::IncludeState) { + if (up->isBool()) up->deserialize(include_state); + else skip_value(*up); + } + else if (key == Key::ReqCompress) { + if (up->isBool()) up->deserialize(compress); + else skip_value(*up); } else skip_value(*up); } @@ -776,16 +1060,20 @@ namespace RNS { namespace Provisioning { size_t applied_total = 0; bool any_reboot = false; + // The set of namespaces that were actually touched by this commit — + // used when IncludeState is true to scope the post-commit Values map. + // If no explicit NamespaceFilter, we collect namespaces that had at + // least one draft to commit (rather than dumping every namespace's + // working state). + std::vector touched_ns; + auto do_one = [&](Namespace& ns) { // Pre-commit hook: fires once per namespace iff at least one // draft entry exists, before any field setter runs. Mirrors // the in-process Provisioner::commit path so wire-driven commits // see identical semantics. - bool has_any_draft = false; - for (const Field& f : ns.fields()) { - if (ns.has_draft(f.id)) { has_any_draft = true; break; } - } - if (has_any_draft && ns.has_on_commit()) { + const bool had_draft = ns.has_any_draft(); + if (had_draft && ns.has_on_commit()) { try { ns.on_commit_callback()(ns); } catch (const std::exception& e) { ERRORF("Provisioning::op_commit: on_commit callback for namespace %u threw: %s", @@ -805,6 +1093,9 @@ namespace RNS { namespace Provisioning { } } if (_storage) _storage->save_namespace(ns); + // Track for the post-commit Values response: include namespaces + // the caller filtered on OR namespaces that actually had drafts. + if (include_state && (has_filter || had_draft)) touched_ns.push_back(&ns); }; if (has_filter) { @@ -818,10 +1109,27 @@ namespace RNS { namespace Provisioning { } set_reboot_flag(any_reboot); - return pack_response((opid_t)Op::Commit, seq, [&](MsgPack::Packer& p) { - p.serialize(MsgPack::map_size_t(2)); + return pack_response((opid_t)Op::Commit, seq, compress, [&](MsgPack::Packer& p) { + // Base entries: Applied + NeedsReboot. IncludeState adds Values + Hash. + const size_t entries = include_state ? 4 : 2; + p.serialize(MsgPack::map_size_t(entries)); p.serialize((uint16_t)Key::Applied); p.serialize((uint64_t)applied_total); p.serialize((uint16_t)Key::NeedsReboot); p.serialize((bool)_needs_reboot); + if (include_state) { + // Build the Values sub-payload into a scratch so we can hash + // it before emitting — matches the GetState convention where + // the Hash is CRC32 over the packed Values bytes (excluding + // the Hash entry itself). PostOpValues/PostOpHash slots avoid + // clashing with Applied's numeric slot 1. + MsgPack::Packer scratch; + pack_ns_values(scratch, touched_ns); + const uint32_t hash = Utilities::Crc::crc32(0, + (const uint8_t*)scratch.data(), scratch.size()); + p.serialize((uint16_t)Key::PostOpValues); + pack_ns_values(p, touched_ns); + p.serialize((uint16_t)Key::PostOpHash); + p.serialize((uint32_t)hash); + } }); } diff --git a/test/test_provisioning/test_provisioning.cpp b/test/test_provisioning/test_provisioning.cpp index 5cb4dfd8..c532edfa 100644 --- a/test/test_provisioning/test_provisioning.cpp +++ b/test/test_provisioning/test_provisioning.cpp @@ -175,6 +175,51 @@ static Bytes make_request(uint8_t op, uint64_t seq, F&& pack_payload) { return Bytes(p.data(), p.size()); } +// Consume one msgpack value at the unpacker's current cursor. Type dispatch +// mirrors what the server-side skip_value does, but Unity-safe and header- +// only so any wire test can use it. +static void t_skip_value(MsgPack::Unpacker& u) { + if (u.isNil()) { MsgPack::object::nil_t n; u.deserialize(n); } + else if (u.isBool()) { bool b; u.deserialize(b); } + else if (u.isUInt() || u.isInt()) { int64_t iv; u.deserialize(iv); } + else if (u.isFloat32() || u.isFloat64()) { double d; u.deserialize(d); } + else if (u.isStr()) { MsgPack::str_t s; u.deserialize(s); } + else if (u.isBin()) { MsgPack::bin_t b; u.deserialize(b); } + else if (u.isArray()) { + const size_t n = u.unpackArraySize(); + for (size_t i = 0; i < n; ++i) t_skip_value(u); + } + else if (u.isMap()) { + const size_t n = u.unpackMapSize(); + for (size_t i = 0; i < n; ++i) { t_skip_value(u); t_skip_value(u); } + } +} + +// Consumes the GetState response envelope up to (but not including) the +// contents of the Values sub-map. Returns the number of namespaces in +// Values with the cursor positioned to read (ns_id, ns_field_map) pairs. +// +// Assumes Values is the first entry in the body — matches the server's +// current emission order. Callers that need to inspect Drafts/Hash/Unchanged +// (which follow Values) re-parse the response from scratch; the helper +// short-circuits at Values to keep the common path simple. +static size_t enter_get_state_values(MsgPack::Unpacker& u, uint8_t expected_op, + uint64_t* seq_out = nullptr) +{ + u.unpackArraySize(); + uint8_t op = 0; u.deserialize(op); + TEST_ASSERT_EQUAL(expected_op, op); + uint64_t seq = 0; u.deserialize(seq); + if (seq_out) *seq_out = seq; + TEST_ASSERT_TRUE(u.isMap()); + u.unpackMapSize(); + + int64_t key = 0; u.deserialize(key); + TEST_ASSERT_EQUAL(Key::Values, key); + TEST_ASSERT_TRUE(u.isMap()); + return u.unpackMapSize(); +} + // --------------------------------------------------------------------------- // Tests // --------------------------------------------------------------------------- @@ -607,15 +652,10 @@ void test_transport_identity_excluded_from_get_state(void) { }); Bytes resp = p.handle_message(req); - // Parse: response is [op, seq, payload-map { ns_id -> { field_id -> value } }] + // Parse: response body = { Values: { ns_id -> { field_id -> value } }, Hash: ... } MsgPack::Unpacker u; u.feed(resp.data(), resp.size()); - u.unpackArraySize(); - uint8_t op = 0; u.deserialize(op); - uint64_t seq = 0; u.deserialize(seq); - TEST_ASSERT_EQUAL((uint8_t)Op::GetState, op); - TEST_ASSERT_TRUE(u.isMap()); - const size_t n_ns = u.unpackMapSize(); + const size_t n_ns = enter_get_state_values(u, (uint8_t)Op::GetState); TEST_ASSERT_EQUAL_size_t(1, n_ns); int64_t ns_id_raw = 0; u.deserialize(ns_id_raw); @@ -794,13 +834,10 @@ void test_command_excluded_from_get_state(void) { Bytes resp = p.handle_message(req); MsgPack::Unpacker u; u.feed(resp.data(), resp.size()); - u.unpackArraySize(); - uint8_t op = 0; u.deserialize(op); - uint64_t seq = 0; u.deserialize(seq); - TEST_ASSERT_TRUE(u.isMap()); - const size_t n_ns = u.unpackMapSize(); + const size_t n_ns = enter_get_state_values(u, (uint8_t)Op::GetState); TEST_ASSERT_EQUAL_size_t(1, n_ns); int64_t ns_id_raw = 0; u.deserialize(ns_id_raw); + (void)ns_id_raw; TEST_ASSERT_TRUE(u.isMap()); const size_t n_fields = u.unpackMapSize(); @@ -871,7 +908,9 @@ void test_command_void_wire_set_state_with_nil(void) { auto& p = Provisioner::instance(); Bytes set_req = make_request((uint8_t)Op::SetState, 1, [&](MsgPack::Packer& pk) { - // Payload is the bare {ns_id: {field_id: value}} map. + // Payload is a request envelope { State: { ns_id: { fid: value } } }. + pk.serialize(MsgPack::map_size_t(1)); + pk.serialize((uint16_t)Key::State); pk.serialize(MsgPack::map_size_t(1)); pk.serialize((uint16_t)CUSTOM_NS_ID); pk.serialize(MsgPack::map_size_t(1)); @@ -1069,11 +1108,7 @@ void test_hierarchy_get_state_stays_flat(void) { Bytes resp = p.handle_message(req); MsgPack::Unpacker u; u.feed(resp.data(), resp.size()); - u.unpackArraySize(); - uint8_t op; u.deserialize(op); - uint64_t seq; u.deserialize(seq); - TEST_ASSERT_TRUE(u.isMap()); - const size_t n_ns = u.unpackMapSize(); + const size_t n_ns = enter_get_state_values(u, (uint8_t)Op::GetState); TEST_ASSERT_EQUAL_size_t(1, n_ns); int64_t top_key = 0; u.deserialize(top_key); TEST_ASSERT_EQUAL(HIER_LORA, top_key); // leaf id at top — NOT parent id @@ -1374,8 +1409,10 @@ void test_wire_set_state_then_commit(void) { fresh_provisioning(g_test_root); auto& p = Provisioner::instance(); - // SetState: { CUSTOM_NS_ID: { CUSTOM_INT: 88 } } + // SetState envelope: { State: { CUSTOM_NS_ID: { CUSTOM_INT: 88 } } } Bytes req = make_request((uint8_t)Op::SetState, 1, [&](MsgPack::Packer& pk) { + pk.serialize(MsgPack::map_size_t(1)); + pk.serialize((uint16_t)Key::State); pk.serialize(MsgPack::map_size_t(1)); pk.serialize((uint16_t)CUSTOM_NS_ID); pk.serialize(MsgPack::map_size_t(1)); @@ -1405,6 +1442,8 @@ void test_wire_set_state_constraint_error(void) { auto& p = Provisioner::instance(); Bytes req = make_request((uint8_t)Op::SetState, 1, [&](MsgPack::Packer& pk) { + pk.serialize(MsgPack::map_size_t(1)); + pk.serialize((uint16_t)Key::State); pk.serialize(MsgPack::map_size_t(1)); pk.serialize((uint16_t)CUSTOM_NS_ID); pk.serialize(MsgPack::map_size_t(1)); @@ -1466,6 +1505,613 @@ void test_wire_get_capabilities(void) { TEST_ASSERT_TRUE(u.isArray()); const size_t cap = u.unpackArraySize(); TEST_ASSERT_GREATER_OR_EQUAL(3, cap); // reticulum + transport + identity + custom + + // Each entry is a map { NsId, NsName, NsParent, NsFieldCount, NsSchemaHash }. + // Walk them all and prove the custom namespace's per-ns hash matches what + // Namespace::schema_hash() reports directly — the wire and the in-memory + // cache must agree. + bool found_custom = false; + uint32_t wire_custom_hash = 0; + uint64_t wire_custom_fcount = 0; + for (size_t i = 0; i < cap; ++i) { + TEST_ASSERT_TRUE(u.isMap()); + const size_t nkeys = u.unpackMapSize(); + int64_t ns_id = 0; + std::string ns_name; + int64_t ns_parent = -1; + int64_t fcount = -1; + int64_t schema_hash = -1; + for (size_t j = 0; j < nkeys; ++j) { + int64_t key = 0; u.deserialize(key); + if (key == Key::NsId) { u.deserialize(ns_id); } + else if (key == Key::NsName) { MsgPack::str_t s; u.deserialize(s); ns_name = std::string(s.c_str()); } + else if (key == Key::NsParent) { u.deserialize(ns_parent); } + else if (key == Key::NsFieldCount) { u.deserialize(fcount); } + else if (key == Key::NsSchemaHash) { u.deserialize(schema_hash); } + else { t_skip_value(u); } + } + if ((uint16_t)ns_id == CUSTOM_NS_ID) { + found_custom = true; + wire_custom_hash = (uint32_t)schema_hash; + wire_custom_fcount = (uint64_t)fcount; + TEST_ASSERT_EQUAL(0, ns_parent); // custom ns is a root + TEST_ASSERT_EQUAL_STRING("custom", ns_name.c_str()); + } + } + TEST_ASSERT_TRUE(found_custom); + TEST_ASSERT_GREATER_THAN(0, wire_custom_fcount); + // The wire hash must match what schema_hash() reports in memory. + const Namespace* ns = p.registry().find((uint16_t)CUSTOM_NS_ID); + TEST_ASSERT_NOT_NULL(ns); + TEST_ASSERT_EQUAL_UINT32(ns->schema_hash(), wire_custom_hash); + TEST_ASSERT_EQUAL_UINT64(ns->fields().size(), wire_custom_fcount); +} + +// --------------------------------------------------------------------------- +// New wire tests for the split/hashed GetState and filtered GetSchema +// --------------------------------------------------------------------------- + +// GetState without Draft flag, no drafts staged: body has Values and Hash, +// no Drafts key. +void test_wire_get_state_working_no_draft(void) { + fresh_provisioning(g_test_root); + auto& p = Provisioner::instance(); + + Bytes req = make_request((uint8_t)Op::GetState, 1, [&](MsgPack::Packer& pk) { + pk.serialize(MsgPack::map_size_t(1)); + pk.serialize((uint16_t)Key::NamespaceFilter); + pk.serialize(MsgPack::arr_size_t(1)); + pk.serialize((uint16_t)CUSTOM_NS_ID); + }); + Bytes resp = p.handle_message(req); + + MsgPack::Unpacker u; u.feed(resp.data(), resp.size()); + u.unpackArraySize(); + uint8_t op = 0; u.deserialize(op); + uint64_t seq = 0; u.deserialize(seq); + TEST_ASSERT_EQUAL((uint8_t)Op::GetState, op); + TEST_ASSERT_TRUE(u.isMap()); + const size_t body_entries = u.unpackMapSize(); + + bool saw_values = false, saw_hash = false, saw_drafts = false; + for (size_t i = 0; i < body_entries; ++i) { + int64_t key = 0; u.deserialize(key); + if (key == Key::Values) { saw_values = true; t_skip_value(u); } + else if (key == Key::Hash) { saw_hash = true; int64_t h; u.deserialize(h); TEST_ASSERT_NOT_EQUAL(0, h); } + else if (key == Key::Drafts) { saw_drafts = true; t_skip_value(u); } + else { t_skip_value(u); } + } + TEST_ASSERT_TRUE(saw_values); + TEST_ASSERT_TRUE(saw_hash); + TEST_ASSERT_FALSE(saw_drafts); + (void)p; +} + +// GetState WITHOUT the Draft flag, drafts staged: response still holds only +// working values. Drafts must not leak into the response when the client +// didn't ask for them. +void test_wire_get_state_working_with_draft(void) { + fresh_provisioning(g_test_root); + auto& p = Provisioner::instance(); + TEST_ASSERT_TRUE(p.field(CUSTOM_NS_ID, CUSTOM_INT, Value((int64_t)77))); + + Bytes req = make_request((uint8_t)Op::GetState, 1, [&](MsgPack::Packer& pk) { + pk.serialize(MsgPack::map_size_t(1)); + pk.serialize((uint16_t)Key::NamespaceFilter); + pk.serialize(MsgPack::arr_size_t(1)); + pk.serialize((uint16_t)CUSTOM_NS_ID); + }); + Bytes resp = p.handle_message(req); + + MsgPack::Unpacker u; u.feed(resp.data(), resp.size()); + const size_t n_ns = enter_get_state_values(u, (uint8_t)Op::GetState); + TEST_ASSERT_EQUAL_size_t(1, n_ns); + // Confirm the working value (not the draft) is returned. + int64_t ns_id = 0; u.deserialize(ns_id); + TEST_ASSERT_EQUAL(CUSTOM_NS_ID, ns_id); + TEST_ASSERT_TRUE(u.isMap()); + const size_t n_f = u.unpackMapSize(); + bool int_seen = false; + for (size_t i = 0; i < n_f; ++i) { + int64_t fid = 0; u.deserialize(fid); + if (fid == CUSTOM_INT) { + int64_t v = 0; u.deserialize(v); + int_seen = true; + TEST_ASSERT_EQUAL_INT64(5, v); // working, not draft (77) + } + else t_skip_value(u); + } + TEST_ASSERT_TRUE(int_seen); + + // Re-parse to verify Drafts key is absent. + MsgPack::Unpacker u2; u2.feed(resp.data(), resp.size()); + u2.unpackArraySize(); + uint8_t op2 = 0; u2.deserialize(op2); + uint64_t seq2 = 0; u2.deserialize(seq2); + TEST_ASSERT_TRUE(u2.isMap()); + const size_t body_entries = u2.unpackMapSize(); + bool saw_drafts = false; + for (size_t i = 0; i < body_entries; ++i) { + int64_t key = 0; u2.deserialize(key); + if (key == Key::Drafts) saw_drafts = true; + t_skip_value(u2); + } + TEST_ASSERT_FALSE(saw_drafts); +} + +// GetState with Draft: true and a draft staged. Body carries Values (working +// state) AND Drafts (sparse map of the drafted fields), delivered in one +// round-trip. The Drafts map is scoped to namespaces that actually have +// drafts and contains only the drafted fields (not the full working set). +void test_wire_get_state_merged_with_drafts(void) { + fresh_provisioning(g_test_root); + auto& p = Provisioner::instance(); + TEST_ASSERT_TRUE(p.field(CUSTOM_NS_ID, CUSTOM_INT, Value((int64_t)77))); + + Bytes req = make_request((uint8_t)Op::GetState, 1, [&](MsgPack::Packer& pk) { + pk.serialize(MsgPack::map_size_t(2)); + pk.serialize((uint16_t)Key::NamespaceFilter); + pk.serialize(MsgPack::arr_size_t(1)); + pk.serialize((uint16_t)CUSTOM_NS_ID); + pk.serialize((uint16_t)Key::Draft); + pk.serialize((bool)true); + }); + Bytes resp = p.handle_message(req); + + // First pass: walk the whole body checking both Values and Drafts keys. + MsgPack::Unpacker u; u.feed(resp.data(), resp.size()); + u.unpackArraySize(); + uint8_t op = 0; u.deserialize(op); + uint64_t seq = 0; u.deserialize(seq); + TEST_ASSERT_EQUAL((uint8_t)Op::GetState, op); + TEST_ASSERT_TRUE(u.isMap()); + const size_t body_entries = u.unpackMapSize(); + + bool saw_values = false, saw_drafts = false, saw_hash = false; + int64_t working_int = -1, draft_int = -1; + int64_t drafts_ns_id = -1; + size_t drafts_field_count = 0; + for (size_t i = 0; i < body_entries; ++i) { + int64_t key = 0; u.deserialize(key); + if (key == Key::Values) { + saw_values = true; + TEST_ASSERT_TRUE(u.isMap()); + const size_t nns = u.unpackMapSize(); + TEST_ASSERT_EQUAL_size_t(1, nns); + int64_t nsid = 0; u.deserialize(nsid); + TEST_ASSERT_EQUAL(CUSTOM_NS_ID, nsid); + TEST_ASSERT_TRUE(u.isMap()); + const size_t nf = u.unpackMapSize(); + for (size_t j = 0; j < nf; ++j) { + int64_t fid = 0; u.deserialize(fid); + if (fid == CUSTOM_INT) { u.deserialize(working_int); } + else t_skip_value(u); + } + } + else if (key == Key::Drafts) { + saw_drafts = true; + TEST_ASSERT_TRUE(u.isMap()); + const size_t nns = u.unpackMapSize(); + TEST_ASSERT_EQUAL_size_t(1, nns); + u.deserialize(drafts_ns_id); + TEST_ASSERT_TRUE(u.isMap()); + drafts_field_count = u.unpackMapSize(); + for (size_t j = 0; j < drafts_field_count; ++j) { + int64_t fid = 0; u.deserialize(fid); + if (fid == CUSTOM_INT) u.deserialize(draft_int); + else t_skip_value(u); + } + } + else if (key == Key::Hash) { + saw_hash = true; + int64_t h = 0; u.deserialize(h); + TEST_ASSERT_NOT_EQUAL(0, h); + } + else t_skip_value(u); + } + TEST_ASSERT_TRUE(saw_values); + TEST_ASSERT_TRUE(saw_drafts); + TEST_ASSERT_TRUE(saw_hash); + TEST_ASSERT_EQUAL_INT64(5, working_int); // working value untouched + TEST_ASSERT_EQUAL_INT64(77, draft_int); // draft value + TEST_ASSERT_EQUAL_INT64(CUSTOM_NS_ID, drafts_ns_id); + TEST_ASSERT_EQUAL_size_t(1, drafts_field_count); // only CUSTOM_INT is drafted +} + +// Draft: true with no drafts staged: server omits the Drafts key entirely +// (same on-wire shape as a working-only response). +void test_wire_get_state_draft_flag_empty(void) { + fresh_provisioning(g_test_root); + auto& p = Provisioner::instance(); + + Bytes req = make_request((uint8_t)Op::GetState, 1, [&](MsgPack::Packer& pk) { + pk.serialize(MsgPack::map_size_t(2)); + pk.serialize((uint16_t)Key::NamespaceFilter); + pk.serialize(MsgPack::arr_size_t(1)); + pk.serialize((uint16_t)CUSTOM_NS_ID); + pk.serialize((uint16_t)Key::Draft); + pk.serialize((bool)true); + }); + Bytes resp = p.handle_message(req); + + MsgPack::Unpacker u; u.feed(resp.data(), resp.size()); + u.unpackArraySize(); + uint8_t op = 0; u.deserialize(op); + uint64_t seq = 0; u.deserialize(seq); + TEST_ASSERT_TRUE(u.isMap()); + const size_t body_entries = u.unpackMapSize(); + bool saw_drafts = false; + for (size_t i = 0; i < body_entries; ++i) { + int64_t key = 0; u.deserialize(key); + if (key == Key::Drafts) saw_drafts = true; + t_skip_value(u); + } + TEST_ASSERT_FALSE(saw_drafts); + (void)p; +} + +// PriorHash cache hit: the second identical GetState request carrying the +// first response's Hash gets { Unchanged: true }. +void test_wire_get_state_prior_hash_hit(void) { + fresh_provisioning(g_test_root); + auto& p = Provisioner::instance(); + + auto make_req = [](uint32_t prior_hash, bool with_prior) { + return make_request((uint8_t)Op::GetState, 1, [&](MsgPack::Packer& pk) { + pk.serialize(MsgPack::map_size_t(with_prior ? 2 : 1)); + pk.serialize((uint16_t)Key::NamespaceFilter); + pk.serialize(MsgPack::arr_size_t(1)); + pk.serialize((uint16_t)CUSTOM_NS_ID); + if (with_prior) { + pk.serialize((uint16_t)Key::PriorHash); + pk.serialize((uint32_t)prior_hash); + } + }); + }; + + // First: no prior hash — server returns full body + Hash. Full-body + // pass to extract Hash and confirm the body wasn't the Unchanged form. + Bytes resp1 = p.handle_message(make_req(0, false)); + uint32_t hash1 = 0; + bool unchanged1 = false; + { + MsgPack::Unpacker u; u.feed(resp1.data(), resp1.size()); + u.unpackArraySize(); + uint8_t op; u.deserialize(op); + uint64_t seq; u.deserialize(seq); + TEST_ASSERT_TRUE(u.isMap()); + const size_t body_entries = u.unpackMapSize(); + for (size_t i = 0; i < body_entries; ++i) { + int64_t key = 0; u.deserialize(key); + if (key == Key::Hash) { int64_t h = 0; u.deserialize(h); hash1 = (uint32_t)h; } + else if (key == Key::Unchanged) { u.deserialize(unchanged1); } + else { t_skip_value(u); } + } + } + TEST_ASSERT_NOT_EQUAL(0, hash1); + TEST_ASSERT_FALSE(unchanged1); + + // Second: send hash1 as PriorHash — server short-circuits to Unchanged. + Bytes resp2 = p.handle_message(make_req(hash1, true)); + bool unchanged2 = false; + { + MsgPack::Unpacker u; u.feed(resp2.data(), resp2.size()); + u.unpackArraySize(); + uint8_t op = 0; u.deserialize(op); + uint64_t seq = 0; u.deserialize(seq); + TEST_ASSERT_TRUE(u.isMap()); + const size_t body_entries = u.unpackMapSize(); + TEST_ASSERT_EQUAL_size_t(1, body_entries); + int64_t key = 0; u.deserialize(key); + TEST_ASSERT_EQUAL(Key::Unchanged, key); + u.deserialize(unchanged2); + } + TEST_ASSERT_TRUE(unchanged2); +} + +// PriorHash cache miss after a commit: the same prior hash no longer +// matches, and the server returns a fresh Hash different from the first. +void test_wire_get_state_prior_hash_miss_after_commit(void) { + fresh_provisioning(g_test_root); + auto& p = Provisioner::instance(); + + auto do_fetch = [&](uint32_t prior, bool with_prior, uint32_t& hash_out, bool& unchanged_out) { + Bytes req = make_request((uint8_t)Op::GetState, 1, [&](MsgPack::Packer& pk) { + pk.serialize(MsgPack::map_size_t(with_prior ? 2 : 1)); + pk.serialize((uint16_t)Key::NamespaceFilter); + pk.serialize(MsgPack::arr_size_t(1)); + pk.serialize((uint16_t)CUSTOM_NS_ID); + if (with_prior) { + pk.serialize((uint16_t)Key::PriorHash); + pk.serialize((uint32_t)prior); + } + }); + Bytes resp = p.handle_message(req); + hash_out = 0; unchanged_out = false; + MsgPack::Unpacker u; u.feed(resp.data(), resp.size()); + u.unpackArraySize(); + uint8_t op; u.deserialize(op); + uint64_t seq; u.deserialize(seq); + TEST_ASSERT_TRUE(u.isMap()); + const size_t body_entries = u.unpackMapSize(); + for (size_t i = 0; i < body_entries; ++i) { + int64_t key = 0; u.deserialize(key); + if (key == Key::Hash) { int64_t h = 0; u.deserialize(h); hash_out = (uint32_t)h; } + else if (key == Key::Unchanged) { u.deserialize(unchanged_out); } + else { t_skip_value(u); } + } + }; + + uint32_t hash1 = 0; bool unchanged1 = false; + do_fetch(0, false, hash1, unchanged1); + TEST_ASSERT_NOT_EQUAL(0, hash1); + TEST_ASSERT_FALSE(unchanged1); + + // Mutate committed state. + TEST_ASSERT_TRUE(p.field(CUSTOM_NS_ID, CUSTOM_INT, Value((int64_t)42))); + TEST_ASSERT_TRUE(p.commit()); + + // Same prior hash — now stale. + uint32_t hash2 = 0; bool unchanged2 = false; + do_fetch(hash1, true, hash2, unchanged2); + TEST_ASSERT_FALSE(unchanged2); + TEST_ASSERT_NOT_EQUAL(0, hash2); + TEST_ASSERT_NOT_EQUAL(hash1, hash2); +} + +// GetSchema with NamespaceFilter must return only the requested namespaces +// (in registry order); the unfiltered case still returns everything. +void test_wire_get_schema_filtered(void) { + fresh_provisioning(g_test_root); + auto& p = Provisioner::instance(); + + // Full fetch: baseline count. + Bytes req_full = make_request((uint8_t)Op::GetSchema, 1, [](MsgPack::Packer& pk) { + MsgPack::object::nil_t n; pk.serialize(n); + }); + Bytes resp_full = p.handle_message(req_full); + size_t full_count = 0; + { + MsgPack::Unpacker u; u.feed(resp_full.data(), resp_full.size()); + u.unpackArraySize(); + uint8_t op; u.deserialize(op); + uint64_t seq; u.deserialize(seq); + TEST_ASSERT_TRUE(u.isArray()); + full_count = u.unpackArraySize(); + } + TEST_ASSERT_GREATER_THAN(1, full_count); + + // Filtered fetch: exactly one namespace requested. + Bytes req_filt = make_request((uint8_t)Op::GetSchema, 2, [&](MsgPack::Packer& pk) { + pk.serialize(MsgPack::map_size_t(1)); + pk.serialize((uint16_t)Key::NamespaceFilter); + pk.serialize(MsgPack::arr_size_t(1)); + pk.serialize((uint16_t)CUSTOM_NS_ID); + }); + Bytes resp_filt = p.handle_message(req_filt); + MsgPack::Unpacker u; u.feed(resp_filt.data(), resp_filt.size()); + u.unpackArraySize(); + uint8_t op; u.deserialize(op); + uint64_t seq; u.deserialize(seq); + TEST_ASSERT_EQUAL((uint8_t)Op::GetSchema, op); + TEST_ASSERT_TRUE(u.isArray()); + const size_t filt_count = u.unpackArraySize(); + TEST_ASSERT_EQUAL_size_t(1, filt_count); + + // The single entry is a 4-tuple [id, name, parent, fields]. + u.unpackArraySize(); + int64_t ns_id = 0; u.deserialize(ns_id); + TEST_ASSERT_EQUAL(CUSTOM_NS_ID, ns_id); +} + +// SetState with IncludeState: true returns the updated Values + Drafts + Hash +// so the client can refresh the panel without a follow-up GetState round-trip. +// SetState changes drafts only — Values reflects working (unchanged), Drafts +// carries the newly-staged entries. +void test_wire_set_state_include_state(void) { + fresh_provisioning(g_test_root); + auto& p = Provisioner::instance(); + + Bytes set_req = make_request((uint8_t)Op::SetState, 1, [&](MsgPack::Packer& pk) { + pk.serialize(MsgPack::map_size_t(2)); + pk.serialize((uint16_t)Key::State); + pk.serialize(MsgPack::map_size_t(1)); + pk.serialize((uint16_t)CUSTOM_NS_ID); + pk.serialize(MsgPack::map_size_t(1)); + pk.serialize((uint16_t)CUSTOM_INT); + pk.serialize((int64_t)88); + pk.serialize((uint16_t)Key::IncludeState); + pk.serialize((bool)true); + }); + Bytes resp = p.handle_message(set_req); + + MsgPack::Unpacker u; u.feed(resp.data(), resp.size()); + u.unpackArraySize(); + uint8_t op = 0; u.deserialize(op); + uint64_t seq = 0; u.deserialize(seq); + TEST_ASSERT_EQUAL((uint8_t)Op::SetState, op); + TEST_ASSERT_TRUE(u.isMap()); + const size_t body_entries = u.unpackMapSize(); + + int64_t applied = -1; + bool saw_values = false, saw_drafts = false, saw_hash = false; + int64_t working_int = -1, draft_int = -1; + bool saw_applied = false; + for (size_t i = 0; i < body_entries; ++i) { + int64_t key = 0; u.deserialize(key); + if (key == Key::Applied && !saw_applied) { u.deserialize(applied); saw_applied = true; } + else if (key == Key::PostOpHash) { saw_hash = true; int64_t h; u.deserialize(h); TEST_ASSERT_NOT_EQUAL(0, h); } + else if (key == Key::PostOpValues) { + saw_values = true; + TEST_ASSERT_TRUE(u.isMap()); + const size_t nns = u.unpackMapSize(); + TEST_ASSERT_EQUAL_size_t(1, nns); + int64_t nsid = 0; u.deserialize(nsid); + TEST_ASSERT_EQUAL(CUSTOM_NS_ID, nsid); + TEST_ASSERT_TRUE(u.isMap()); + const size_t nf = u.unpackMapSize(); + for (size_t j = 0; j < nf; ++j) { + int64_t fid = 0; u.deserialize(fid); + if (fid == CUSTOM_INT) u.deserialize(working_int); + else t_skip_value(u); + } + } + else if (key == Key::PostOpDrafts) { + saw_drafts = true; + TEST_ASSERT_TRUE(u.isMap()); + const size_t nns = u.unpackMapSize(); + TEST_ASSERT_EQUAL_size_t(1, nns); // only the touched namespace with drafts + int64_t nsid = 0; u.deserialize(nsid); + TEST_ASSERT_EQUAL(CUSTOM_NS_ID, nsid); + TEST_ASSERT_TRUE(u.isMap()); + const size_t nf = u.unpackMapSize(); + TEST_ASSERT_EQUAL_size_t(1, nf); // only CUSTOM_INT is drafted + int64_t fid = 0; u.deserialize(fid); + TEST_ASSERT_EQUAL(CUSTOM_INT, fid); + u.deserialize(draft_int); + } + else t_skip_value(u); + } + TEST_ASSERT_EQUAL_INT64(1, applied); + TEST_ASSERT_TRUE(saw_values); + TEST_ASSERT_TRUE(saw_drafts); + TEST_ASSERT_TRUE(saw_hash); + TEST_ASSERT_EQUAL_INT64(5, working_int); // working unchanged + TEST_ASSERT_EQUAL_INT64(88, draft_int); // draft staged + (void)p; +} + +// Commit with IncludeState: true returns the post-commit Values plus a Hash, +// eliminating the follow-up GetState round-trip. Verify the applied value +// shows up in Values and Hash is present. +void test_wire_commit_include_state(void) { + fresh_provisioning(g_test_root); + auto& p = Provisioner::instance(); + + // Stage a draft via SetState. + Bytes set_req = make_request((uint8_t)Op::SetState, 1, [&](MsgPack::Packer& pk) { + pk.serialize(MsgPack::map_size_t(1)); + pk.serialize((uint16_t)Key::State); + pk.serialize(MsgPack::map_size_t(1)); + pk.serialize((uint16_t)CUSTOM_NS_ID); + pk.serialize(MsgPack::map_size_t(1)); + pk.serialize((uint16_t)CUSTOM_INT); + pk.serialize((int64_t)88); + }); + (void)p.handle_message(set_req); + + // Commit with IncludeState=true. + Bytes commit_req = make_request((uint8_t)Op::Commit, 2, [&](MsgPack::Packer& pk) { + pk.serialize(MsgPack::map_size_t(1)); + pk.serialize((uint16_t)Key::IncludeState); + pk.serialize((bool)true); + }); + Bytes resp = p.handle_message(commit_req); + + MsgPack::Unpacker u; u.feed(resp.data(), resp.size()); + u.unpackArraySize(); + uint8_t op = 0; u.deserialize(op); + uint64_t seq = 0; u.deserialize(seq); + TEST_ASSERT_EQUAL((uint8_t)Op::Commit, op); + TEST_ASSERT_TRUE(u.isMap()); + const size_t body_entries = u.unpackMapSize(); + + int64_t applied = -1; + bool needs_reboot = false; + bool saw_values = false, saw_hash = false; + int64_t applied_value = -1; + // Commit response uses PostOpValues/PostOpHash slots (distinct from + // Applied's slot 1, which happens to numerically equal Key::Values). + bool saw_applied = false; + for (size_t i = 0; i < body_entries; ++i) { + int64_t key = 0; u.deserialize(key); + if (key == Key::Applied && !saw_applied) { u.deserialize(applied); saw_applied = true; } + else if (key == Key::NeedsReboot) { u.deserialize(needs_reboot); } + else if (key == Key::PostOpHash) { saw_hash = true; int64_t h; u.deserialize(h); TEST_ASSERT_NOT_EQUAL(0, h); } + else if (key == Key::PostOpValues) { + saw_values = true; + TEST_ASSERT_TRUE(u.isMap()); + const size_t nns = u.unpackMapSize(); + TEST_ASSERT_EQUAL_size_t(1, nns); // only the namespace that had a draft + int64_t nsid = 0; u.deserialize(nsid); + TEST_ASSERT_EQUAL(CUSTOM_NS_ID, nsid); + TEST_ASSERT_TRUE(u.isMap()); + const size_t nf = u.unpackMapSize(); + for (size_t j = 0; j < nf; ++j) { + int64_t fid = 0; u.deserialize(fid); + if (fid == CUSTOM_INT) u.deserialize(applied_value); + else t_skip_value(u); + } + } + else t_skip_value(u); + } + TEST_ASSERT_EQUAL_INT64(1, applied); + TEST_ASSERT_FALSE(needs_reboot); + TEST_ASSERT_TRUE(saw_values); + TEST_ASSERT_TRUE(saw_hash); + TEST_ASSERT_EQUAL_INT64(88, applied_value); // value the client just committed + (void)p; +} + +// Commit without IncludeState returns just {Applied, NeedsReboot} — the +// pre-Phase-3 shape. Verify no Values or Hash leak in. +void test_wire_commit_without_include_state(void) { + fresh_provisioning(g_test_root); + auto& p = Provisioner::instance(); + + Bytes set_req = make_request((uint8_t)Op::SetState, 1, [&](MsgPack::Packer& pk) { + pk.serialize(MsgPack::map_size_t(1)); + pk.serialize((uint16_t)Key::State); + pk.serialize(MsgPack::map_size_t(1)); + pk.serialize((uint16_t)CUSTOM_NS_ID); + pk.serialize(MsgPack::map_size_t(1)); + pk.serialize((uint16_t)CUSTOM_INT); + pk.serialize((int64_t)88); + }); + (void)p.handle_message(set_req); + + Bytes commit_req = make_request((uint8_t)Op::Commit, 2, [](MsgPack::Packer& pk) { + MsgPack::object::nil_t n; pk.serialize(n); + }); + Bytes resp = p.handle_message(commit_req); + + MsgPack::Unpacker u; u.feed(resp.data(), resp.size()); + u.unpackArraySize(); + uint8_t op = 0; u.deserialize(op); + uint64_t seq = 0; u.deserialize(seq); + TEST_ASSERT_TRUE(u.isMap()); + const size_t body_entries = u.unpackMapSize(); + TEST_ASSERT_EQUAL_size_t(2, body_entries); // exactly Applied + NeedsReboot + bool saw_values = false, saw_hash = false; + // Skip the first two entries (Applied, NeedsReboot); we're looking for + // any PostOpValues/PostOpHash entries beyond them. + for (size_t i = 0; i < body_entries; ++i) { + int64_t key = 0; u.deserialize(key); + if (i >= 2) { + if (key == Key::PostOpValues) saw_values = true; + else if (key == Key::PostOpHash) saw_hash = true; + } + t_skip_value(u); + } + TEST_ASSERT_FALSE(saw_values); + TEST_ASSERT_FALSE(saw_hash); + (void)p; +} + +// Per-namespace schema hash must be stable across begin() cycles as long as +// the schema itself is unchanged. Sanity check against silent hash drift. +void test_namespace_schema_hash_stable(void) { + fresh_provisioning(g_test_root); + const Namespace* ns = Provisioner::instance().registry().find((uint16_t)CUSTOM_NS_ID); + TEST_ASSERT_NOT_NULL(ns); + const uint32_t hash_boot_1 = ns->schema_hash(); + TEST_ASSERT_NOT_EQUAL(0, hash_boot_1); + + // Full re-init: end(), re-register the exact same schema, begin() again. + fresh_provisioning(g_test_root); + const Namespace* ns2 = Provisioner::instance().registry().find((uint16_t)CUSTOM_NS_ID); + TEST_ASSERT_NOT_NULL(ns2); + TEST_ASSERT_EQUAL_UINT32(hash_boot_1, ns2->schema_hash()); } void test_wire_factory_reset(void) { @@ -1650,6 +2296,17 @@ int runUnityTests(void) { RUN_TEST(test_wire_set_state_then_commit); RUN_TEST(test_wire_set_state_constraint_error); RUN_TEST(test_wire_get_capabilities); + RUN_TEST(test_wire_set_state_include_state); + RUN_TEST(test_wire_get_state_working_no_draft); + RUN_TEST(test_wire_get_state_working_with_draft); + RUN_TEST(test_wire_get_state_merged_with_drafts); + RUN_TEST(test_wire_get_state_draft_flag_empty); + RUN_TEST(test_wire_get_state_prior_hash_hit); + RUN_TEST(test_wire_get_state_prior_hash_miss_after_commit); + RUN_TEST(test_wire_get_schema_filtered); + RUN_TEST(test_wire_commit_include_state); + RUN_TEST(test_wire_commit_without_include_state); + RUN_TEST(test_namespace_schema_hash_stable); RUN_TEST(test_wire_factory_reset); RUN_TEST(test_wire_reboot); RUN_TEST(test_wire_reboot_no_callback_still_acks);