Skip to content

Add portable record export across surfaces - #122

Open
tony wants to merge 71 commits into
issue-80-deterministic-identitiesfrom
issue-81-export
Open

Add portable record export across surfaces#122
tony wants to merge 71 commits into
issue-80-deterministic-identitiesfrom
issue-81-export

Conversation

@tony

@tony tony commented Jul 12, 2026

Copy link
Copy Markdown
Owner

Summary

Add the initial portable export contract across CLI, TUI, and MCP using one deterministic, frontend-neutral renderer.

  • Add agentgrep export for stable NDJSON or Markdown, bodies on by default, standard output or explicit files, bounded selection, no-clobber output, and explicit --force.
  • Add HUD /export [PATH] and /export-thread [PATH] for one selected record or its observed thread, with private collision-free defaults and all identity, rendering, and disk work off the Textual message pump.
  • Add a read-only export_records MCP tool for one to 20 existing search refs, bodies opt-in, a 400 KiB inline artifact ceiling, bounded opaque refs, and no server-side filesystem authority.
  • Share one exact allowlist that excludes source paths, origins, titles, session identifiers, and arbitrary metadata while preserving canonical IDs and an artifact-level schema version.
  • Harden source-alias protection, atomic writes, phase-error privacy, legacy ref compatibility, duplicate selection handling, PATH_MAX ref round-trips, and docs/runtime schema parity.

Relationship

This PR targets the branch for #111 because deterministic record identities are its prerequisite. It is independent of the bookmark PR and can be retargeted to master after #111 lands.

Closes #81

Tested

  • Full repository gate: Ruff, formatting, ty, 2,891 passed, 3 skipped, 3 expected failures, and Sphinx.
  • Format × body × cardinality × selection permutations; CLI stdout/file/force/refusal; MCP size/ref/privacy boundaries; TUI races, reset, teardown, large payloads, and static non-blocking guards.
  • Live Textual pane: private and explicit Markdown writes, literal bracketed filenames, no-overwrite refusal, preserved results, private modes, unchanged source stores, responsive narrow layout, and clean exit.

Goal

Idempotent completion condition for /goal. It asserts an end state, not an action: if the state already holds, the first evaluation passes and nothing is edited.

/goal PR #122 is green on its base and its ADR stops claiming a number. Work ONLY in the worktree agentgrep-issue-81-export; check out no branch in the shared master tree; never push. Report FIRST: git rev-parse --abbrev-ref HEAD (prints issue-81-export; judge local HEAD, not a remote ref), git -c rerere.enabled=false merge-tree --write-tree --name-only issue-80-deterministic-identities HEAD, NO_COLOR=1 AGENTGREP_TUI_WATCHDOG=1 PATH=$PWD/.venv/bin:$PATH env -u VIRTUAL_ENV -u UV_NO_SYNC ./.venv/bin/python -m pytest -m '' --reruns 0 -q, uv run ruff check ., uv run ty check, and grep -A3 '^## Status' docs/dev/adr/*portable-record-export.md. End state: merge-tree prints zero CONFLICT lines against that LOCAL base tip as it stands, the other three exit 0, and that Status line records the ADR's NUMBER as provisional pending merge-time allocation. If that tip moved (#111 may rebase it), rebase onto it once rather than waiting on or reverting anything. Those three pass today, so the Status line is the work; renumber and reslug nothing, numbering being a merge-time call. Without the PATH prefix the shell resolves agentgrep to the master checkout, which lacks the export verb, so doc examples fail. Non-goals: no rebase onto master, no PR retarget, no landing #111. Preserve ADR 0011: identity, rendering, scanning, path validation and disk writes stay off the pump, with no scandir, lstat, open, sqlite, subprocess, network, json load/dump or whole-artifact CPU reachable from a pump callable a hop down (the hook sees neither, so trace by hand), pump entrypoints @pump_only, workers @offload. Keep schema_version and the format value extensible; call no shipped format set, ceiling or allowlist permanent; let export deny no index or provider. Or stop after 12 turns and report what remains.
  • Proofgit rev-parse --abbrev-ref HEAD && git -c rerere.enabled=false merge-tree --write-tree --name-only issue-80-deterministic-identities HEAD && grep -A3 '^## Status' docs/dev/adr/*portable-record-export.md && NO_COLOR=1 AGENTGREP_TUI_WATCHDOG=1 PATH=$PWD/.venv/bin:$PATH env -u VIRTUAL_ENV -u UV_NO_SYNC ./.venv/bin/python -m pytest -m '' --reruns 0 -q
  • Idempotence — Once the Status line carries the provisional-number note the proof is purely read-only: the branch still merges clean into its local base tip and the suite still passes, so a re-run changes nothing; a base tip moved by Expose deterministic IDs and tighten TUI controls #111 costs exactly one rebase and then the same steady state.
  • Blocked by — nothing (landing Expose deterministic IDs and tighten TUI controls #111 is an explicit non-goal; the branch is judged against its own local base tip as it currently stands)
  • Blocks — nothing
  • Preserves — The export format value, inline ceiling and field allowlist stay versioned and extensible so csv/html/mermaid and vendor messages profiles remain addable; export makes no claim that search reads only live stores, leaving Durable prompt corpus and derived search indexes (ADR 0019) #142's corpus and Persistent SQLite DB index with cache-aware search #47's index provider open; ADR 0011 placement of identity, rendering, scandir/lstat and disk writes off the pump is retained; the ADR number stays available to the other branch claiming it.

@tony
tony force-pushed the issue-80-deterministic-identities branch from adcf5e0 to cb6377d Compare July 12, 2026 20:14
@tony
tony force-pushed the issue-81-export branch from 59d247d to 1be23b6 Compare July 12, 2026 20:15
@tony
tony force-pushed the issue-80-deterministic-identities branch 9 times, most recently from a68c683 to 5ffd718 Compare July 14, 2026 03:56
@tony
tony force-pushed the issue-81-export branch from 1be23b6 to 89c6db2 Compare July 14, 2026 04:28
@tony
tony force-pushed the issue-80-deterministic-identities branch from 5ffd718 to 4e49bbd Compare July 14, 2026 23:36
@tony
tony force-pushed the issue-81-export branch from 1f4ca31 to 5c30da7 Compare July 14, 2026 23:36
@tony
tony force-pushed the issue-80-deterministic-identities branch from 4e49bbd to c755ba0 Compare July 18, 2026 11:40
@tony
tony force-pushed the issue-81-export branch from 5c30da7 to 5daffab Compare July 18, 2026 12:05
@tony
tony force-pushed the issue-81-export branch from 77e86c3 to 6cc123a Compare July 18, 2026 13:51
@tony
tony force-pushed the issue-80-deterministic-identities branch from c755ba0 to a0c10d6 Compare July 19, 2026 01:06
@tony
tony force-pushed the issue-81-export branch from 6cc123a to a8566ef Compare July 19, 2026 01:19
@tony
tony force-pushed the issue-80-deterministic-identities branch from a0c10d6 to 68db741 Compare July 19, 2026 01:42
@tony
tony force-pushed the issue-81-export branch from a8566ef to 07f4877 Compare July 19, 2026 01:45
tony added 28 commits August 9, 2026 07:30
why: Directory completion delegated tilde expansion to process-global account
lookup instead of the home already owned by the TUI session.

what:
- Pass the session home from the export dialog through the picker worker.
- Resolve only current-user tilde syntax with the shared safe resolver.
- Cover session-home expansion and other-user rejection without expanduser.
why: Canceling a Textual worker cannot stop a thread already blocked in
scandir, so rapid edits could overlap filesystem enumerations.

what:
- Keep one active enumeration and coalesce newer input into one latest slot.
- Launch the queued scan only after the blocked filesystem call returns.
- Preserve debounce, generation, focus, and unmount invalidation contracts.
why: Opening an untrusted preference path with buffered I/O could block startup
on a FIFO and accepted non-regular file types before parsing.

what:
- Open preference files nonblocking without following the final symlink.
- Require a regular fstat result before one bounded read loop.
- Cover FIFO latency, path-free fallback, and the exact 16 KiB boundary.
why: Unicode format controls can reorder or hide characters in a reviewed
basename without appearing as ordinary control characters.

what:
- Reject category Cf in template literals and compiled filenames.
- Cover zero-width and bidi override controls with focused regressions.
why: A clean first-use session remembered an app-owned export path that the
review validator required to exist, so the default flow could not proceed.

what:
- Create only the computed private default through the descriptor-safe walker.
- Keep arbitrary missing and symlinked directories validation-only and rejected.
- Prove clean-home export, 0700 mode, and path safety in Pilot regressions.
why: Pressing y accepted the durable action while the disabled confirmation
list still visibly highlighted No.

what:
- Move the review highlight to Save before disabling confirmation.
- Pin the visible saving state in the existing single-write Pilot test.
why: The inline template error fell below a 30 by 10 tmux viewport, leaving a
focused editor with no visible explanation.

what:
- Make edit and review stages vertically scrollable only when space requires it.
- Scroll inline errors into view without adding pump-side work.
- Cover feedback and stage reachability at 40x12, 30x10, and 60x16.
why: The TUI guide still described e as an immediate private export and blurred
its remembered review state with one-shot slash commands.

what:
- Document the reviewed directory, template, exact filename, and No path.
- Keep explicit and private-default slash-command behavior distinct.
- Scope the documentation contract to the Export section.
why: A failed reviewed save restored its retained inline draft and also emitted
a duplicate toast carrying the same error.

what:
- Let a mounted saving dialog consume its worker failure inline.
- Preserve path-free error toasts for slash-command and no-dialog failures.
- Cover retained drafts, cancel keys, and the no-dialog writer route.
why: Absolute export directories inside the active session home exposed a machine-specific prefix in the edit and review stages and in remembered preferences.

what:
- Compact session-home directories to tilde drafts without resolving paths.
- Apply compaction to loaded and newly entered directory values.
- Cover defaults, legacy preferences, outside paths, and persistence.
why: Ctrl-C dismissed the export modal immediately while editing, making a familiar terminal clear-field gesture unnecessarily destructive.

what:
- Clear a focused non-empty directory or template before dismissal.
- Preserve focus after clearing and dismiss on a second Ctrl-C.
- Retain immediate review cancellation and saving protection.
why: Restoring an asynchronous export failure at compact terminal sizes let deferred focus scrolling move the sole inline feedback outside the viewport.

what:
- Restore retained editor focus before applying asynchronous feedback.
- Reveal the error through a named callback after refreshed layout.
- Cover retained focus and visibility at 30 by 10.
why: A post-refresh error reveal could outlive its edit state and scroll a dialog after rapid dismissal or after its feedback had been cleared.

what:
- Scope deferred reveals to a generated non-empty error request.
- Require the mounted active dialog to remain in its edit phase.
- Invalidate pending reveals on updates, review, dismissal, and unmount.
- Cover rapid Escape and cleared-error races deterministically.
why: Directory drafts could contain control, bidi-format, or surrogate code points that made reviewed destinations and completion rows ambiguous.

what:
- Share one reviewability rule across preference load, save, and submit.
- Omit unsafe existing directory names from bounded completion.
- Cover control, invisible, bidi, surrogate, config, and dialog cases.
why: Draft validation created the first-use export directory before the user accepted Save, so review, No, and cancel crossed an unexpected mutation boundary.

what:
- Validate a missing exact app default through a read-only no-symlink prefix walk.
- Create that directory securely only in the accepted export worker.
- Keep arbitrary missing and symlinked paths rejected.
- Document artifact and private-preference mutation boundaries.
why: The widgets package re-exported completion rows, popup chrome, and export workflow values that are internal module details rather than reusable widgets.

what:
- Keep internal candidate, popup, draft, and intent types out of package exports.
- Import the intent type directly from its defining module.
- Pin the narrower package boundary without making an API claim.
why: The export changelog used an ambiguous pronoun and implied that every input state handled the shortcut identically.

what:
- State directly that e remains ordinary text when an input is focused.
- Pin the exact focused-input wording in the export documentation contract.
why: An empty directory draft resolved to the process working directory, so review could display a blank value while Save targeted an implicit location.

what:
- Reject empty text in the shared directory validator while preserving literal whitespace-only Unix paths.
- Cover resolution, preference load and save, dialog review, and completion boundaries.
why: Lexical directory compaction runs on the Textual pump, where an unbounded path with many components could stall input handling before validation was offloaded.

what:
- Apply a documented 4,096-character ceiling to the live editor and shared validation and persistence boundaries.
- Reject oversized Enter snapshots and compaction inputs before path processing, with boundary regressions.
why: Export editing should follow the directional muscle memory of the rest of the TUI, while review should remain calm and legible in small terminals.

what:
- Add clamped Ctrl-H/J/K/L and completion-aware arrow traversal.
- Pin edit and review hints without hiding validation feedback.
- Shrink-wrap a No-first confirmation with a moving selection marker.
- Cover navigation, layout, saving, and pump roles with Pilot tests.
why: Modal export obscured the active explorer and made the reviewed save
flow hard to distinguish from the reader. A pane-owned state machine keeps
context visible and gives shortcut and slash-command entry one behavior.

what:
- Replace the selected-record modal with a fresh detail-pane export flow.
- Restore the exact search draft, selection, focus, zoom, and reader state.
- Keep validation and durable output off the Textual message pump.
- Document and test the reviewed /export contract and compact layout.
why: Leading-slash text that is not a registered command is a valid search.
Treating every slash-prefixed value as transient restored an older query after
the next successful command.

what:
- Remember the active input after dispatch classifies it as a literal query.
- Cover exact value and selection restoration after a leading-slash search.
why: A second Tab could leave the export state for the live search input,
allowing hidden searches whose results no longer matched the restored query.

what:
- Wrap Tab and Shift-Tab across export editors and the review choice.
- Preserve Tab acceptance for an open directory completion.
- Cover edit and compact-review focus containment with Pilot.
why: Textual mounts dynamically added children after the opening pump turn.
A same-turn priority key could query export controls before compose completed.

what:
- Track when the pane's composed controls are ready for routed actions.
- Ignore edit, cancel, and review actions during the deferred mount window.
- Reproduce the race without an intervening Pilot pause.
why: The export action should replace the detail reader on wide terminals,
not obscure the results that give the action its selection context.

what:
- Preserve the results column beside the export pane in wide layouts.
- Keep the focused full-body export presentation for stacked terminals.
- Cover responsive transitions without changing reader state.
why: Only the direct thread command uses an automatic canonical-ID name;
the reviewed record command previews and confirms a templated filename.

what:
- Attribute automatic private naming specifically to /export-thread.
- Pin the command distinction in the export documentation contract.
why: Reusing the FastMCP decorator mutates function metadata shared with
the docs shim, making schema results depend on registration order.

what:
- Build dedicated Tool objects when registering docs schema functions.
- Keep the live export docstring and documented schema text aligned.
why: Private Textual reactive hooks couple query-draft state to
framework internals and lose cursor-only selections.

what:
- Observe value and selection through public watch registrations.
- Guard the public pump callbacks and cursor-only restoration behavior.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant