Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,21 @@ Two readers exist: `claude_code` and `codex`. Adding a third means adding a `Rea
- **Redaction** (`redact.rs`) replaces credential-shaped substrings before storage and counts them per kind in the envelope. Best-effort, not a guarantee — it catches machine-issued token shapes, not a password typed in prose. The real defense is the default filtering above. `harness::home_dir()` reads `$HOME` directly rather than via the `dirs` crate: a **testability decision**, since every harness path derives from it and a test that sets `HOME` to a tempdir is then hermetic by construction. Do not turn it into a dependency.
- **Fixtures are synthetic**, never copied sessions (`tests/fixtures/transcripts/README.md` explains why): a real transcript holds whatever the user pasted.

## Importing exported transcripts (`engram import`)

`src/import.rs`. `ingest` reads the session file a harness writes for itself, which covers a harness engram has a reader for and nothing else. `import` reads what was *exported* — so it reaches a harness whose store engram cannot parse (protobuf, editor workspace state) but which offers its own export command, and it reads back engram's own `save-chat` archives, which were previously the one format engram could write and not read.

- **Deliberately not a `ReaderKind`.** That enum is built around "one path per installed harness", discovered from a `HarnessSpec` plus a working directory. A file somebody hands you fits none of that, so `import` feeds the shared pipeline directly (`normalize_text` → `redact::scrub` → `Store::ingest_turns`) without touching `TranscriptSupport`.
- **Identity is the content, not the file.** `import_id` is a v5 uuid over `(scope, agent, role, created_at, text)`, so every dedupe requirement falls out of `INSERT OR IGNORE`: re-importing inserts 0, thirteen byte-identical copies of one file import once, and the archive holding two concatenated copies of itself (residue of the old appending `save-chat`) collapses to one set. No separate dedupe pass exists, deliberately.
- **Three formats, sniffed by content not extension.** engram Texinfo (both dialects — current `@chapter`, legacy `@section` — one heading grammar covers all 7,163 messages in the real corpus), Opencode Markdown, and Claude Code scrollback. A real `chat/` dir held a standalone HTML palette editor; extension-matching would have ingested it. Source directories merely *named* `chat` (Kotlin, TypeScript, Rust) are skipped the same way. Unrecognised files are reported in `skipped` **with a reason** — the difference between "found nothing" and "could not read nine files".
- **A structural line is never message content.** Escaping doubles every literal `@`, so any line starting with a single `@` is markup. Without that filter the trailing `@c Signed by:` / `@chapter Chat history…` of one document became the tail of the previous message's body — which is precisely why the double-appended archive failed to deduplicate at first: the last message of each copy differed by the *next* copy's header.
- **`--input-format`, not `--format`.** `--format` is the global output-format flag; a subcommand reusing the name makes clap panic on the duplicate argument id (`Mismatch between definition and access`), not shadow it.
- **Canonicalize before resolving scope.** `import ./chat` resolves its parent to `.`, whose basename is empty, and the scope silently became `default` — one relative path from filing a whole corpus under the wrong name.
- **Roles are not binary.** The corpus held `assistant` 6395, `user` 632, **`note` 136**. The third is carried through, not coerced.
- **Missing timestamps are synthesised in order, never from the clock.** Markdown and scrollback record no per-message time. Each message is offset one second from a file-level anchor (the export's recorded session time, else the file's mtime); `approximate_times` is reported per file. Stamping `now()` would collapse a conversation into one instant and destroy reading order — the same failure `transcript` refuses a wall-clock fallback to avoid.
- **The walker must not honour `.gitignore`.** engram itself adds `chat/` to a project's `.gitignore` on first `save-chat`, so every archive it has written is ignored by definition; `fd`, `rg` and `git ls-files` all find zero of them by default. `std::fs` ignores ignore-rules, which is the only reason this command works.
- **Known unhandled variants**, both real and both reported as skips: older Claude Code scrollback with no `❯` prompt marker (`--input-format claude-scrollback` recovers the `●` assistant turns only), and Goose terminal output (`> ` marks the user).

## Harness command delivery (`engram install`)

`src/install.rs` + `plugins/engram/`. Engram was already an MCP server in every harness on a typical machine; what was missing was a *command surface*.
Expand Down
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,14 @@ follows [Keep a Changelog](https://keepachangelog.com/); versions follow

### Added

- **`engram import`** — reads chat transcripts that were exported to files, for
the harnesses `ingest` has no reader for, and reads back `save-chat`'s own
`.texi` archives. Three formats, detected by content: engram Texinfo (both
dialects), Opencode Markdown, and Claude Code scrollback. Each file is filed
into the scope of the project containing it; identity is a v5 uuid over the
message's content, so re-importing inserts nothing and duplicate files
collapse. `--input-format` forces a parser, `--recursive` descends.

- **VS Code** and **Cursor** are supported harnesses (the tenth and eleventh).
VS Code gets `~/.config/Code/User/prompts/engram-<name>.prompt.md`; Cursor
gets `~/.cursor/skills/engram-<name>/SKILL.md`. VS Code's `mcp.json` keys its
Expand Down
65 changes: 65 additions & 0 deletions doc/engram.texi
Original file line number Diff line number Diff line change
Expand Up @@ -827,6 +827,71 @@ Transcripts can be very large --- one rollout on a developer machine was
@code{--max-bytes} (64@tie{}MiB by default) refuses an oversized
transcript with an error naming the override rather than attempting it.

@section engram import

@cindex import command
@cindex exported transcripts
@cindex chat archives

@example
engram import @var{path}@dots{} [--scope @var{id}] [--input-format @var{fmt}]
[--recursive] [--max-bytes @var{n}] [--dry-run]
@end example

Reads chat transcripts that were @emph{exported} to files, rather than the
session file a harness writes for itself. @command{engram ingest} covers a
harness engram has a reader for; this covers everything else. Two cases
motivate it. A harness may keep its history in a form engram cannot parse
--- protocol buffers, an editor's workspace state --- while still offering
its own export command. And an archive is not a live session at all:
@command{engram save-chat} could write a @file{.texi} that engram had no way
to read back, so engram's own output was the one format it could not ingest.

Three formats are recognised, by @emph{content} rather than by extension:
archives written by @command{save-chat} (both dialects, the current
@code{@@chapter} form and the older @code{@@section} one), Opencode's
Markdown export, and raw terminal scrollback. A file that announces none of
them is skipped with a reason rather than guessed at --- a real @file{chat/}
directory held a standalone HTML palette editor alongside genuine archives,
and source files in directories that merely happen to be named @file{chat}
are skipped the same way. @code{--input-format} forces a parser for a file
that cannot be detected. It is spelled that way because @code{--format} is
the global output-format option.

@cindex scope, on import
Each file is filed into the scope of the project @emph{containing} it, so one
run over many directories fills many scopes correctly. The scope recorded
inside an archive is deliberately ignored: in practice those strings are
inconsistent --- bare names, relative paths, absolute paths and branch names
all appear --- while the containing project is unambiguous.

@cindex idempotence, of import
A message's identity is a version@tie{}5 UUID over its scope, agent, role,
timestamp and text, so every deduplication requirement falls out of
@code{INSERT OR IGNORE} rather than a separate pass. Re-importing an archive
inserts nothing. A file copied into four projects imports once per scope
instead of four times. An archive holding two concatenated copies of itself
--- the residue of an earlier @command{save-chat} that appended rather than
rewrote --- collapses back to one set of messages.

@cindex roles, third
Roles are not binary. A real corpus of 7,163 archived messages held 6,395
@code{assistant}, 632 @code{user} and 136 @code{note}; the third is carried
through rather than coerced into one of the other two.

@cindex timestamps, synthesised
The Markdown and scrollback formats record no per-message time. Rather than
refuse them or stamp them from the clock --- which would collapse a whole
conversation into one instant and destroy reading order, the failure
@command{engram ingest} refuses a wall-clock fallback to avoid --- each
message is offset one second from a file-level anchor: the session time the
export records, else the file's own modification time. Absolute values are
approximate and the response says so per file, in
@code{approximate_times}; the ordering is exact.

Text is normalized and redacted on the way in, exactly as a live transcript
is. An archive is not more trustworthy for being old.

@section engram install

@cindex install command
Expand Down
34 changes: 34 additions & 0 deletions src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -347,6 +347,40 @@ EXAMPLES:
#[arg(long)]
dry_run: bool,
},
/// Import chat transcripts that were exported to files.
///
/// Reads what a harness's own export command wrote, and what `save-chat`
/// archived, for the harnesses engram has no reader for.
Import {
Comment on lines +350 to +354

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Advertise the import command in the capability manifest

Adding this CLI command without updating Command::Describe leaves import absent from both the manifest's commands list and its mcp.cli_only list. Clients using the JSON capability manifest for discovery will conclude that this command does not exist, so the new surface must be added to that manifest alongside the other CLI-only commands.

AGENTS.md reference: AGENTS.md:L38-L38

Useful? React with 👍 / 👎.

/// Files or directories to import. A directory is scanned for exports.
#[arg(value_name = "PATH", required = true)]
paths: Vec<std::path::PathBuf>,

/// Scope to store into. Resolves from each file's own project when
/// omitted, so one run can fill many scopes correctly.
#[arg(long)]
scope: Option<String>,

/// Force a parser instead of detecting one from the file's content.
///
/// Deliberately not `--format`: that is the global output-format flag,
/// and giving a subcommand the same name makes clap panic on the
/// duplicate argument id rather than shadow it.
#[arg(long, value_enum)]
input_format: Option<crate::import::Format>,

/// Descend into subdirectories.
#[arg(long)]
recursive: bool,

/// Refuse an export larger than this many bytes.
#[arg(long, default_value_t = crate::transcript::DEFAULT_MAX_BYTES)]
max_bytes: u64,

/// Parse and report what would be stored, without writing.
#[arg(long)]
dry_run: bool,
},
/// Write engram's slash commands into the harnesses on this machine.
///
/// Engram is usually already registered as an MCP server everywhere; what
Expand Down
Loading
Loading