Expose deterministic IDs and tighten TUI controls - #111
Open
tony wants to merge 33 commits into
Open
Conversation
tony
force-pushed
the
issue-80-deterministic-identities
branch
from
July 12, 2026 11:06
6e41258 to
ce9b300
Compare
tony
force-pushed
the
issue-80-deterministic-identities
branch
from
July 12, 2026 12:30
ce9b300 to
adcf5e0
Compare
This was referenced Jul 12, 2026
tony
force-pushed
the
issue-80-deterministic-identities
branch
from
July 12, 2026 20:14
adcf5e0 to
cb6377d
Compare
tony
force-pushed
the
issue-80-deterministic-identities
branch
from
July 12, 2026 23:16
4bb78dd to
88fa234
Compare
tony
force-pushed
the
issue-80-deterministic-identities
branch
from
July 12, 2026 23:41
114a66f to
7b27c8f
Compare
tony
force-pushed
the
issue-80-deterministic-identities
branch
from
July 13, 2026 01:41
f5d4530 to
06dfe7d
Compare
tony
force-pushed
the
issue-80-deterministic-identities
branch
from
July 13, 2026 23:05
b4724de to
619a9af
Compare
tony
force-pushed
the
issue-80-deterministic-identities
branch
from
July 13, 2026 23:51
619a9af to
eef6ae2
Compare
tony
force-pushed
the
issue-80-deterministic-identities
branch
from
July 14, 2026 00:09
eef6ae2 to
c667bd6
Compare
why: Stable machine-readable handles let downstream features reference logical records without changing physical inspect refs. what: - Add flat content, record, stability, and thread fields to search rows. - Require the nullable identity siblings in MCP output schemas. - Reuse prepared digests while preserving ref and cursor bytes. - Share position-aware tuple dedupe across every execution driver.
why: Stable handles in ranked search and the detail pane let users cite records without running identity work on the Textual message pump. what: - Show full record, content, and thread handles in ranked text and HUD detail. - Prepare HUD identity and large bodies in one generation-gated worker. - Bound object-safe caches while preserving live body and find renderables.
why: Stable content, occurrence, and thread handles need one public contract so users and downstream features do not confuse identity with physical result resolution or inferred conversation completeness. what: - Add ADR 0014 with canonical vectors, dedupe forms, topology limits, ref compatibility, and privacy boundaries. - Document the shipped CLI, TUI, and MCP identity shapes and add the issue 80 changelog deliverable. - Add typed documentation contract tests for every public surface.
why: Source-local ordinals were compared across adapters, and legacy session records without canonical namespaces collapsed by path. what: - Scope ordinal IDs, dedupe keys, and conversation ordering to store/adapter domains. - Preserve explicit legacy sessions across both execution drivers and bounded source accounting. - Add forward-only topology to Codex and Grok aggregate prompt histories. - Document coordinate-domain and fallback-thread semantics.
why: Threadless inputs hashed bodies before rejection. Inventory ties preserved caller order for public record fields. what: - Reject missing threads before content hashing. - Reuse retained thread identifiers. - Complete inventory keys across public scalars, origin, position, and nested metadata. - Add permutation and 1 MiB hashing-budget regressions.
why: Unnamespaced thread coordinates are source-local, so treating them as logical IDs collapsed records from separate stores and paths. what: - Add a fixed source-thread dedupe form for unnamespaced anchors. - Keep logical and fallback forms exclusive to namespaced anchors. - Compare cheap inventory metadata before exact record text. - Pin driver, bounded-scan, ordering, and ADR behavior with tests.
why: Source-order adapters pass enumerate() ordinals through the coordinate validator. Normalizing two absent native coordinates per record adds measurable work to large histories. what: - Return the same source-order position before native normalization when both native coordinates are absent. - Cover the fast path while keeping validation tests for native and malformed coordinates.
why: Exhaustive adapter and narrow-terminal permutations exposed explicit source anchors that remained null and HUD identity rows that wrapped or cropped after resize. what: - Preserve defensible thread and occurrence coordinates across default and inspectable adapters while keeping ambiguous fallbacks null. - Keep full HUD handles on one row with responsive labels and a resize-only header refresh. - Add regressions for adapter topology, narrow rendering, and detail find row accounting.
why: Distinct stored occurrences could share a position-blind physical fingerprint, so inspect_result stopped at the first match and returned the wrong canonical occurrence. what: - Fold validated native or source-order coordinates into current search ref fingerprints without changing token shape or length. - Match historical position-blind fingerprints as a bounded fallback. - Document compatibility and cover repeated, threadless, and malformed coordinate cases.
why: Permissive base64 decoding discarded non-alphabet characters, and Python's integer hierarchy let JSON booleans pass as token versions, offsets, and limits. what: - Decode refs and cursors with CPython's validating base64url path. - Reject booleans at every JSON integer boundary. - Cover malformed refs and both cursor kinds while preserving valid version 1 bytes.
why: Cursor transcripts can store identical turns at different raw positions. Adapter-level content deduplication erased that occurrence distinction before deterministic record IDs were assigned. what: - Retain every readable Cursor transcript occurrence. - Let raw source ordinals distinguish otherwise equal stored turns.
why: Latest catalog semantics make encrypted loose Antigravity protobuf stores discoverable only as catalog entries. They cannot supply readable records or deterministic thread anchors. what: - Limit Antigravity UUID anchors to readable CLI stores. - Test valid conversation identity and catalog-only rejection. - Keep duplicate-view coverage on the three readable CLI views.
why: The identity changelog contract checks its reader-facing promise without normalizing Markdown line breaks. The refreshed release prose split that promise across lines. what: - Keep the repeated-content phrase intact in the release entry.
why: Release rollover moved issue 80 from the a34 placeholder into the a35 placeholder. Pinning a changelog test to one prerelease number makes every release commit fail an otherwise current feature branch. what: - Locate the active Yet to be released block by its versioned heading. - Check the lead sentence against the captured current version.
why: The character cap prevents oversized records from freezing Textual, but users need to know that the view is bounded and how to retrieve the unchanged full record. what: - State the line and character limits in the detail renderer contract. - Document CLI JSON and MCP inspection routes to the full body.
why: The palette replacement is discoverable in-app, but the TUI guide still omitted its shared commands, layout-specific zoom targets, and screenshot delivery behavior. what: - Document command-menu discovery in HUD and GrepLog. - List shared commands and their argument forms. - Explain key-panel toggling and clean screenshot delivery.
why: A q binding bypassed the staged Ctrl-C warning and exited directly from results. Grep-log had the same one-key exit path. what: - Share keyed quit confirmation across both TUI layouts. - Cancel pending confirmation on focus, mode, and workflow changes. - Cover results, detail, grep-log, input, and cancellation paths.
why: Draining scans cannot use source-local dedupe counts to stop early. Identity-aware keys were therefore computed in the scanner and again in the global collector. what: - Allocate local dedupe state only for bounded scans with a limit. - Keep key projection at the global frontier for draining scans.
why: Adapter helpers return fresh mutable candidates. Copying each one solely to attach source identity wastes allocations on every parsed row. what: - Attach namespaces and positions directly to fresh candidates. - Verify normalization receives the original enriched candidate.
why: Optimized JSONL scans discarded physical ordinals, so the same native-free occurrence could lose its record ID and repeated equal turns could collapse during deduplication. what: - Carry byte-derived source coordinates through forward, filtered, and reverse JSONL readers. - Apply stable positions across all optimized adapters and document gapped source ordinals. - Cover scan parity, multi-turn lines, exact boundary spans, and bounded execution.
why: Cursor JSONL events can contain multiple message candidates. Reusing the raw line index for each candidate collapsed repeated occurrences into the same record ID. what: - Read Cursor transcripts with the positioned JSONL iterator. - Combine each byte offset with its within-line candidate index. - Cover stable repeated IDs, dedupe, and single-pass parsing.
why: JSONL ordinals are file-local. Comparing them across paths fabricated chronology for records that shared only a thread and adapter. what: - Include the physical path in the private ordering domain. - Withhold linearization across separate source files. - Document that physical topology stays outside canonical IDs.
why: The position-blind fallback only works while every other normalized fingerprint field still matches. A blanket persisted-ref promise overstated this prerelease contract. what: - Describe refs and cursors as snapshot-relative physical handles. - Keep the narrow position-blind first-match fallback. - Direct callers to canonical IDs and fresh refs after changes.
why: The branch already carries a material pi-like TUI control pass. The release entry described only deterministic identity. what: - Name both release outcomes in the lead. - Add a TUI command, focus, screenshot, and quit deliverable. - Keep the two-deliverable release contract under test.
why: Trunk now reserves version-led changelog summaries for release cuts. Feature branches should contribute only scoped deliverable sections. what: - Remove the unreleased a36 summary while retaining both deliverables. - Move the identity ADR link into its deliverable and guard the rule.
tony
force-pushed
the
issue-80-deterministic-identities
branch
from
August 9, 2026 12:04
d9e1601 to
629ed55
Compare
why: Ruff's format gate rejected a focused identity test after the rebase even though lint and type checking passed. what: - Apply Ruff's canonical wrapping to the identity position fixture.
tony
force-pushed
the
issue-80-deterministic-identities
branch
from
August 9, 2026 12:29
629ed55 to
c9e4ed6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add deterministic, privacy-safe identities for normalized search records and
observed conversations, expose them consistently through CLI text, JSON/NDJSON,
MCP, and the HUD TUI, and disclose the accompanying pi-style TUI command pass.
This establishes three distinct versioned handles:
agc1:identifies semantic content from record kind, case-folded role, andexact normalized text.
agr1:identifies one defensible native or source-order occurrence.agt1:identifies one defensible backend-native thread anchor.Each handle uses the first 128 bits of a domain-separated SHA-256 envelope,
encoded as full lowercase base32hex. Unknown record or thread identity remains
null; agentgrep does not invent identity from a path, timestamp, or file mtime.Design
the supported Codex, Claude, Cursor, Antigravity, Gemini, Grok, Pi, OpenCode,
and VS Code stores.
ordinals only when the scan direction proves them stable.
ConversationUnitgrouping with explicitnative_tree,source_order, orunorderedfidelity. It represents only the recordsobserved by the caller; it does not claim source completeness, select a
revision, or invent timestamp chronology.
(store, adapter, path)domain, so file-local ordinals from separate transcripts cannot fabricate a
shared timeline.
forms. Namespaced coordinates may compare across physical views; unnamespaced
coordinates remain scoped to their source path and adapter.
off the Textual message pump, and bound/generation-gate the HUD caches.
agref1:andagcur1:as opaque version 1 tokens. Positioned resultsbind the physical fingerprint to their occurrence coordinate; a position-blind
ref retains bounded first-match fallback only while every other normalized
fingerprint input still matches. Refs and cursors are snapshot-relative
physical handles, while canonical IDs remain comparison handles.
alternate spellings and JSON booleans at integer fields before dispatch.
ADR 0015 records the canonical envelopes, field policy, nullability, dedupe
forms, topology limits, privacy model, physical-handle lifetime, and downstream
contract for bookmarks, exports, and similarity.
TUI controls
menu across HUD and grep-log.
/screenshotSVGcapture, and make help, filtered-row styling, and quit confirmation consistent.
Issue refinement
The issue proposed several useful directions but left important policy forks
open. This PR resolves them conservatively:
single short universal ID with auto-widening.
This gives #79 durable record/thread targets with an explicit content fallback,
requires #81 to own null-thread partitioning and complete-source/revision policy,
and lets #82 use canonical IDs for result/cache equality while using
agref1:to resolve a physical seed.
Compatibility and performance
additive and required-nullable at typed output boundaries.
agcur1:bytes andpositionless
agref1:bytes remain unchanged. Newly emitted positioned refsgain occurrence-aware fingerprints. Physical refs and cursors are
snapshot-relative and may need to be reacquired after store or normalization
changes.
commit orders. No user-visible command crossed the 5% median / 10% p95
investigation threshold in both orders.
MCP consumers hash zero prompt bytes, and 500 threadless 1 MiB records are
rejected before hashing.
benchmark by about 15% in both orders. End-to-end parsing remained order-noisy,
so no broader speedup is claimed.
former iterator, at about 9.5 microseconds per loop.
Verification
The required repository gate passed on the final commit:
$ rm -rf docs/_build; uv run ruff check . --fix --show-fixes; uv run ruff format .; uv run ty check; uv run py.test --reruns 0 -vvv; just build-docs;Results: 2,720 passed, 3 skipped, and 3 expected failures; documentation built
successfully.
The original three-way implementation bakeoff compared position-aware v1
fingerprints, canonical record IDs embedded in refs, and explicit v2 refs.
Adversarial Python, MCP/DX, and terminal/TUI judging selected the position-aware
v1 design 2-1: it resolves repeated turns exactly without longer handles or
coupling physical locators to canonical identity, while retaining a bounded
position-blind fallback.
A follow-up implementation bakeoff compared a localized physical-source repair,
a typed source-coordinate domain, and late identity/dedupe disambiguation.
Adversarial Python, typing, performance, MCP/DX, TUI, concision, and slop reviews
selected the localized repair: it fixes Cursor's within-line coordinates and
bounds conversation ordering for every adapter without expanding public models
or changing canonical IDs.
A live MCP smoke test resolved all seven positioned results from a bounded
ten-result page to their exact record IDs and rejected a malformed ref without
emitting store content.
Independent adversarial reviews also reproduced the source-domain, session,
ordering, threadless-hash, CLI/MCP parity, TUI worker/cache/race, ref-byte,
privacy, import-boundary, and documentation invariants. A watchdog-enabled live
TUI run searched 3,256 real-store matches, resized between 77 and 40 columns,
switched records, filtered to zero and back, and exited cleanly; full identity
handles stayed on one row at both widths.
Closes #80
Goal
Idempotent completion condition for
/goal. It asserts an end state, not an action: if the state already holds, the first evaluation passes and nothing is edited.git rev-parse --abbrev-ref HEAD && git -c rerere.enabled=false merge-tree --write-tree --name-only master HEAD && grep -A3 '^## Status' docs/dev/adr/*deterministic-record-identity.md && NO_COLOR=1 AGENTGREP_TUI_WATCHDOG=1 PATH=$PWD/.venv/bin:$PATH env -u VIRTUAL_ENV -u UV_NO_SYNC ./.venv/bin/python -m pytest -m '' --reruns 0 -q