diff --git a/AGENTS.md b/AGENTS.md index cbb1dcc..c4e6ce7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -267,6 +267,21 @@ Two readers exist: `claude_code` and `codex`. Adding a third means adding a `Rea - **Redaction** (`redact.rs`) replaces credential-shaped substrings before storage and counts them per kind in the envelope. Best-effort, not a guarantee — it catches machine-issued token shapes, not a password typed in prose. The real defense is the default filtering above. `harness::home_dir()` reads `$HOME` directly rather than via the `dirs` crate: a **testability decision**, since every harness path derives from it and a test that sets `HOME` to a tempdir is then hermetic by construction. Do not turn it into a dependency. - **Fixtures are synthetic**, never copied sessions (`tests/fixtures/transcripts/README.md` explains why): a real transcript holds whatever the user pasted. +## Importing exported transcripts (`engram import`) + +`src/import.rs`. `ingest` reads the session file a harness writes for itself, which covers a harness engram has a reader for and nothing else. `import` reads what was *exported* — so it reaches a harness whose store engram cannot parse (protobuf, editor workspace state) but which offers its own export command, and it reads back engram's own `save-chat` archives, which were previously the one format engram could write and not read. + +- **Deliberately not a `ReaderKind`.** That enum is built around "one path per installed harness", discovered from a `HarnessSpec` plus a working directory. A file somebody hands you fits none of that, so `import` feeds the shared pipeline directly (`normalize_text` → `redact::scrub` → `Store::ingest_turns`) without touching `TranscriptSupport`. +- **Identity is the content, not the file.** `import_id` is a v5 uuid over `(scope, agent, role, created_at, text)`, so every dedupe requirement falls out of `INSERT OR IGNORE`: re-importing inserts 0, thirteen byte-identical copies of one file import once, and the archive holding two concatenated copies of itself (residue of the old appending `save-chat`) collapses to one set. No separate dedupe pass exists, deliberately. +- **Three formats, sniffed by content not extension.** engram Texinfo (both dialects — current `@chapter`, legacy `@section` — one heading grammar covers all 7,163 messages in the real corpus), Opencode Markdown, and Claude Code scrollback. A real `chat/` dir held a standalone HTML palette editor; extension-matching would have ingested it. Source directories merely *named* `chat` (Kotlin, TypeScript, Rust) are skipped the same way. Unrecognised files are reported in `skipped` **with a reason** — the difference between "found nothing" and "could not read nine files". +- **A structural line is never message content.** Escaping doubles every literal `@`, so any line starting with a single `@` is markup. Without that filter the trailing `@c Signed by:` / `@chapter Chat history…` of one document became the tail of the previous message's body — which is precisely why the double-appended archive failed to deduplicate at first: the last message of each copy differed by the *next* copy's header. +- **`--input-format`, not `--format`.** `--format` is the global output-format flag; a subcommand reusing the name makes clap panic on the duplicate argument id (`Mismatch between definition and access`), not shadow it. +- **Canonicalize before resolving scope.** `import ./chat` resolves its parent to `.`, whose basename is empty, and the scope silently became `default` — one relative path from filing a whole corpus under the wrong name. +- **Roles are not binary.** The corpus held `assistant` 6395, `user` 632, **`note` 136**. The third is carried through, not coerced. +- **Missing timestamps are synthesised in order, never from the clock.** Markdown and scrollback record no per-message time. Each message is offset one second from a file-level anchor (the export's recorded session time, else the file's mtime); `approximate_times` is reported per file. Stamping `now()` would collapse a conversation into one instant and destroy reading order — the same failure `transcript` refuses a wall-clock fallback to avoid. +- **The walker must not honour `.gitignore`.** engram itself adds `chat/` to a project's `.gitignore` on first `save-chat`, so every archive it has written is ignored by definition; `fd`, `rg` and `git ls-files` all find zero of them by default. `std::fs` ignores ignore-rules, which is the only reason this command works. +- **Known unhandled variants**, both real and both reported as skips: older Claude Code scrollback with no `❯` prompt marker (`--input-format claude-scrollback` recovers the `●` assistant turns only), and Goose terminal output (`> ` marks the user). + ## Harness command delivery (`engram install`) `src/install.rs` + `plugins/engram/`. Engram was already an MCP server in every harness on a typical machine; what was missing was a *command surface*. diff --git a/CHANGELOG.md b/CHANGELOG.md index ae05a13..98cd2ac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,6 +47,14 @@ follows [Keep a Changelog](https://keepachangelog.com/); versions follow ### Added +- **`engram import`** — reads chat transcripts that were exported to files, for + the harnesses `ingest` has no reader for, and reads back `save-chat`'s own + `.texi` archives. Three formats, detected by content: engram Texinfo (both + dialects), Opencode Markdown, and Claude Code scrollback. Each file is filed + into the scope of the project containing it; identity is a v5 uuid over the + message's content, so re-importing inserts nothing and duplicate files + collapse. `--input-format` forces a parser, `--recursive` descends. + - **VS Code** and **Cursor** are supported harnesses (the tenth and eleventh). VS Code gets `~/.config/Code/User/prompts/engram-.prompt.md`; Cursor gets `~/.cursor/skills/engram-/SKILL.md`. VS Code's `mcp.json` keys its diff --git a/doc/engram.texi b/doc/engram.texi index 45e7227..596eae8 100644 --- a/doc/engram.texi +++ b/doc/engram.texi @@ -827,6 +827,71 @@ Transcripts can be very large --- one rollout on a developer machine was @code{--max-bytes} (64@tie{}MiB by default) refuses an oversized transcript with an error naming the override rather than attempting it. +@section engram import + +@cindex import command +@cindex exported transcripts +@cindex chat archives + +@example +engram import @var{path}@dots{} [--scope @var{id}] [--input-format @var{fmt}] + [--recursive] [--max-bytes @var{n}] [--dry-run] +@end example + +Reads chat transcripts that were @emph{exported} to files, rather than the +session file a harness writes for itself. @command{engram ingest} covers a +harness engram has a reader for; this covers everything else. Two cases +motivate it. A harness may keep its history in a form engram cannot parse +--- protocol buffers, an editor's workspace state --- while still offering +its own export command. And an archive is not a live session at all: +@command{engram save-chat} could write a @file{.texi} that engram had no way +to read back, so engram's own output was the one format it could not ingest. + +Three formats are recognised, by @emph{content} rather than by extension: +archives written by @command{save-chat} (both dialects, the current +@code{@@chapter} form and the older @code{@@section} one), Opencode's +Markdown export, and raw terminal scrollback. A file that announces none of +them is skipped with a reason rather than guessed at --- a real @file{chat/} +directory held a standalone HTML palette editor alongside genuine archives, +and source files in directories that merely happen to be named @file{chat} +are skipped the same way. @code{--input-format} forces a parser for a file +that cannot be detected. It is spelled that way because @code{--format} is +the global output-format option. + +@cindex scope, on import +Each file is filed into the scope of the project @emph{containing} it, so one +run over many directories fills many scopes correctly. The scope recorded +inside an archive is deliberately ignored: in practice those strings are +inconsistent --- bare names, relative paths, absolute paths and branch names +all appear --- while the containing project is unambiguous. + +@cindex idempotence, of import +A message's identity is a version@tie{}5 UUID over its scope, agent, role, +timestamp and text, so every deduplication requirement falls out of +@code{INSERT OR IGNORE} rather than a separate pass. Re-importing an archive +inserts nothing. A file copied into four projects imports once per scope +instead of four times. An archive holding two concatenated copies of itself +--- the residue of an earlier @command{save-chat} that appended rather than +rewrote --- collapses back to one set of messages. + +@cindex roles, third +Roles are not binary. A real corpus of 7,163 archived messages held 6,395 +@code{assistant}, 632 @code{user} and 136 @code{note}; the third is carried +through rather than coerced into one of the other two. + +@cindex timestamps, synthesised +The Markdown and scrollback formats record no per-message time. Rather than +refuse them or stamp them from the clock --- which would collapse a whole +conversation into one instant and destroy reading order, the failure +@command{engram ingest} refuses a wall-clock fallback to avoid --- each +message is offset one second from a file-level anchor: the session time the +export records, else the file's own modification time. Absolute values are +approximate and the response says so per file, in +@code{approximate_times}; the ordering is exact. + +Text is normalized and redacted on the way in, exactly as a live transcript +is. An archive is not more trustworthy for being old. + @section engram install @cindex install command diff --git a/src/cli.rs b/src/cli.rs index e2965a2..1562b22 100644 --- a/src/cli.rs +++ b/src/cli.rs @@ -347,6 +347,40 @@ EXAMPLES: #[arg(long)] dry_run: bool, }, + /// Import chat transcripts that were exported to files. + /// + /// Reads what a harness's own export command wrote, and what `save-chat` + /// archived, for the harnesses engram has no reader for. + Import { + /// Files or directories to import. A directory is scanned for exports. + #[arg(value_name = "PATH", required = true)] + paths: Vec, + + /// Scope to store into. Resolves from each file's own project when + /// omitted, so one run can fill many scopes correctly. + #[arg(long)] + scope: Option, + + /// Force a parser instead of detecting one from the file's content. + /// + /// Deliberately not `--format`: that is the global output-format flag, + /// and giving a subcommand the same name makes clap panic on the + /// duplicate argument id rather than shadow it. + #[arg(long, value_enum)] + input_format: Option, + + /// Descend into subdirectories. + #[arg(long)] + recursive: bool, + + /// Refuse an export larger than this many bytes. + #[arg(long, default_value_t = crate::transcript::DEFAULT_MAX_BYTES)] + max_bytes: u64, + + /// Parse and report what would be stored, without writing. + #[arg(long)] + dry_run: bool, + }, /// Write engram's slash commands into the harnesses on this machine. /// /// Engram is usually already registered as an MCP server everywhere; what diff --git a/src/import.rs b/src/import.rs new file mode 100644 index 0000000..b55771c --- /dev/null +++ b/src/import.rs @@ -0,0 +1,684 @@ +// SPDX-FileCopyrightText: 2026 Mohamed Hammad +// SPDX-License-Identifier: GPL-3.0-or-later + +//! Importing chat transcripts that were *exported* rather than read live. +//! +//! [`crate::transcript`] reads the session file a harness writes for itself. +//! That covers a harness engram has a reader for, and nothing else — which +//! left two gaps. A harness may store its history somewhere engram cannot +//! parse (protocol buffers, an editor's workspace state) while still offering +//! its own export command. And an archive written months ago is not a live +//! session at all: `save-chat` could write a `.texi` that engram had no way to +//! read back, so engram's own output was the one format it could not ingest. +//! +//! An export file is the common denominator, so this module takes one and +//! turns it into the same rows `transcript::capture` produces. It is +//! deliberately **not** a [`crate::harness::ReaderKind`]: that enum is built +//! around "one path per installed harness", discovered from a `HarnessSpec` +//! and a working directory, and a file somebody hands you fits none of that. +//! +//! Three properties are worth stating because each was learned from the +//! corpus rather than guessed: +//! +//! * **Identity is the content, not the file.** A message's id is a v5 uuid +//! over `(scope, agent, role, created_at, text)`, so re-importing the same +//! archive inserts nothing, a file copied to four projects imports once per +//! scope rather than four times per scope, and the old `save-chat` append +//! bug — which left one on-disk archive holding two concatenated copies of +//! the same document — collapses back to one set of messages. No separate +//! dedupe pass is needed; `INSERT OR IGNORE` does it. +//! * **Roles are not binary.** A real corpus of 7,163 archived messages held +//! `assistant` 6,395, `user` 632 and **`note` 136**. Coercing that third +//! role into one of the other two would rewrite history to fit a type. +//! * **A missing timestamp is synthesised in order, never invented from the +//! clock.** See [`anchor_series`]. + +use std::path::Path; + +/// An export format engram can parse. +#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize, clap::ValueEnum)] +#[serde(rename_all = "kebab-case")] +#[clap(rename_all = "kebab-case")] +pub enum Format { + /// A `.texi` written by `engram save-chat`, either dialect. + EngramTexinfo, + /// Opencode's (and Kilo's) Markdown export. + OpencodeMarkdown, + /// Raw Claude Code terminal scrollback, pasted into a file. + ClaudeScrollback, +} + +impl Format { + /// The `--format` value and the name reported back. + #[must_use] + pub fn as_str(self) -> &'static str { + match self { + Format::EngramTexinfo => "engram-texinfo", + Format::OpencodeMarkdown => "opencode-markdown", + Format::ClaudeScrollback => "claude-scrollback", + } + } +} + +/// One message recovered from an export. +#[derive(Debug, Clone, serde::Serialize)] +pub struct Message { + pub agent: String, + /// `user`, `assistant` or `note` — whatever the archive recorded. + pub role: String, + pub text: String, + pub created_at: String, + /// True when [`anchor_series`] supplied the timestamp. + pub approximate: bool, +} + +/// Why an import could not proceed. +#[derive(Debug)] +pub enum ImportError { + Io(std::io::Error), + TooLarge { + bytes: u64, + max_bytes: u64, + }, + /// Nothing in the file looked like a transcript engram knows. + Unrecognised, + /// A recognised format whose timestamp did not parse. As in + /// [`crate::transcript`], this is an error rather than a substitution: + /// recall orders by `created_at`, so a wrong value corrupts reading order + /// invisibly. + BadTimestamp { + value: String, + }, +} + +impl std::fmt::Display for ImportError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + ImportError::Io(e) => write!(f, "{e}"), + ImportError::TooLarge { bytes, max_bytes } => { + write!( + f, + "export is {bytes} bytes, above the {max_bytes}-byte ceiling" + ) + } + ImportError::Unrecognised => { + write!(f, "no known export format matched this file") + } + ImportError::BadTimestamp { value } => { + write!(f, "unparseable timestamp: {value}") + } + } + } +} + +/// Identifies the format of an export by looking at its content. +/// +/// Content, not extension, on purpose: the corpus contained two `.html` files +/// in a `chat/` directory that were a standalone palette editor rather than a +/// conversation, and a future `.html` genuinely could be a web export. A file +/// that does not announce itself is skipped rather than guessed at. +#[must_use] +pub fn sniff(text: &str) -> Option { + if text.contains("\\input texinfo") { + return Some(Format::EngramTexinfo); + } + // Opencode's export leads with a `**Session ID:**` block and marks every + // speaker with a level-two heading. + if text.contains("**Session ID:**") + || text + .lines() + .any(|l| l.starts_with("## Assistant (") || l.trim_end() == "## Assistant") + { + return Some(Format::OpencodeMarkdown); + } + // Scrollback has no headings at all — the speaker is a glyph in column 0. + if text.lines().any(|l| l.starts_with('❯')) { + return Some(Format::ClaudeScrollback); + } + None +} + +/// Parses an export whose format is already known. +/// +/// `anchor` is the fallback instant for formats that do not record per-message +/// times; see [`anchor_series`]. +/// +/// # Errors +/// +/// Returns [`ImportError::BadTimestamp`] when a recorded timestamp does not +/// parse. +pub fn parse( + text: &str, + format: Format, + anchor: jiff::Timestamp, +) -> Result, ImportError> { + match format { + Format::EngramTexinfo => parse_texinfo(text), + Format::OpencodeMarkdown => Ok(parse_opencode_markdown(text, anchor)), + Format::ClaudeScrollback => Ok(parse_claude_scrollback(text, anchor)), + } +} + +/// Assigns the `index`-th message of a timestamp-less export its instant. +/// +/// One second per message from a file-level anchor. The absolute values are +/// approximate and say so (`Message::approximate`), but the *order* — the only +/// thing `recall` actually depends on — is exact. The alternative considered +/// and rejected was stamping every message with the same instant, which would +/// collapse a whole conversation into one moment and destroy reading order, +/// the precise failure [`crate::transcript`] refuses a wall-clock fallback to +/// avoid. +#[must_use] +pub fn anchor_series(anchor: jiff::Timestamp, index: usize) -> String { + let shifted = anchor + .checked_add(jiff::Span::new().seconds(i64::try_from(index).unwrap_or(i64::MAX))) + .unwrap_or(anchor); + // `jiff::Timestamp`'s Display is RFC 3339 with a `Z` suffix, which is + // exactly what `normalize_timestamp` produces for the recorded case. + shifted.to_string() +} + +// ------------------------------------------------------------------ texinfo + +/// Reverses [`crate::archive::escape_texinfo`]. +/// +/// That function maps exactly three characters — `@`→`@@`, `{`→`@{`, `}`→`@}` +/// — and nothing else, so this is a three-way reverse and not a general +/// Texinfo decoder. Scanning left to right matters: a naive +/// `replace("@@", "@")` run before the brace rules would turn an escaped +/// `@@{` into `@{` and then into a brace that was never there. +#[must_use] +pub fn unescape_texinfo(s: &str) -> String { + let mut out = String::with_capacity(s.len()); + let mut chars = s.chars(); + while let Some(c) = chars.next() { + if c == '@' { + match chars.next() { + Some(n @ ('@' | '{' | '}')) => out.push(n), + Some(other) => { + out.push('@'); + out.push(other); + } + None => out.push('@'), + } + } else { + out.push(c); + } + } + out +} + +/// Splits a `Message by () at ` heading. +/// +/// Parsed from the right, because an agent name is the only free-form part: +/// the timestamp is the last token, `(role)` the token before it. The two +/// archive dialects differ only in heading level — the current writer emits +/// `@chapter` (the document already has an `@top`), the older one `@section` +/// under a title chapter — so both are accepted and the level carries no +/// meaning beyond that. +fn parse_message_heading(line: &str) -> Option<(String, String, String)> { + let rest = line + .strip_prefix("@chapter Message by ") + .or_else(|| line.strip_prefix("@section Message by "))?; + let (left, ts) = rest.rsplit_once(" at ")?; + let left = left.trim_end(); + let open = left.rfind(" (")?; + let role = left[open + 2..].strip_suffix(')')?; + let agent = &left[..open]; + if agent.is_empty() || role.is_empty() || ts.trim().is_empty() { + return None; + } + Some((agent.to_string(), role.to_string(), ts.trim().to_string())) +} + +/// True for a Texinfo command line, which is never message content. +/// +/// `@@` is an escaped literal `@` and therefore *is* content; a single `@` +/// followed by anything else is markup engram wrote. +fn is_structural(line: &str) -> bool { + let mut chars = line.chars(); + chars.next() == Some('@') && chars.next() != Some('@') +} + +fn parse_texinfo(text: &str) -> Result, ImportError> { + let mut out: Vec = Vec::new(); + let mut current: Option<(String, String, String)> = None; + let mut body = String::new(); + + let flush = |head: Option<(String, String, String)>, + body: &mut String, + out: &mut Vec| + -> Result<(), ImportError> { + let Some((agent, role, ts)) = head else { + body.clear(); + return Ok(()); + }; + let created_at = crate::transcript::normalize_timestamp(&ts) + .map_err(|value| ImportError::BadTimestamp { value })?; + let text = unescape_texinfo(body.trim_end_matches('\n')); + body.clear(); + if text.trim().is_empty() { + return Ok(()); + } + out.push(Message { + agent: unescape_texinfo(&agent), + role, + text, + created_at, + approximate: false, + }); + Ok(()) + }; + + for line in text.lines() { + if let Some(head) = parse_message_heading(line) { + flush(current.take(), &mut body, &mut out)?; + current = Some(head); + continue; + } + if line.trim_end() == "@bye" { + flush(current.take(), &mut body, &mut out)?; + continue; + } + // A structural line can never be message content: escaping doubles + // every literal `@`, so content always arrives as `@@…`. Without this + // the trailing `@c Signed by:` and `@chapter Chat history…` lines of + // one document became the tail of the previous message's body — which + // is exactly how a file holding two concatenated copies of itself + // failed to deduplicate: the last message of each copy differed by the + // header of the next one. + if is_structural(line) { + continue; + } + if current.is_some() { + body.push_str(line); + body.push('\n'); + } + } + flush(current.take(), &mut body, &mut out)?; + Ok(out) +} + +// ----------------------------------------------------------------- markdown + +/// True for a line that opens a message in Opencode's export. +/// +/// Anchored on the two literal speakers rather than on `^## `, because an +/// assistant's own reply routinely contains level-two headings — the observed +/// export had `## Root Cause Analysis`, `## Objective`, `## Next Move` inside +/// message bodies. Splitting on every `## ` shattered one 294 KB conversation +/// of 100 messages into several hundred fragments. +fn markdown_speaker(line: &str) -> Option<&'static str> { + let t = line.trim_end(); + if t == "## User" || t.starts_with("## User (") { + return Some("user"); + } + if t == "## Assistant" || t.starts_with("## Assistant (") { + return Some("assistant"); + } + None +} + +/// Reads `**Created:** 7/7/2026, 9:39:41 PM` out of the export's header. +/// +/// US-locale, not RFC 3339 — the exporter formats for a reader, not for a +/// parser. Returns `None` rather than guessing when the shape does not match, +/// leaving the caller's file-level anchor in place. +fn opencode_created(text: &str) -> Option { + let line = text + .lines() + .take(40) + .find_map(|l| l.trim().strip_prefix("**Created:**"))?; + let (date, time) = line.trim().split_once(", ")?; + let mut parts = date.split('/'); + let month: i8 = parts.next()?.trim().parse().ok()?; + let day: i8 = parts.next()?.trim().parse().ok()?; + let year: i16 = parts.next()?.trim().parse().ok()?; + + let time = time.trim(); + let (clock, meridiem) = time.rsplit_once(' ')?; + let mut hms = clock.split(':'); + let mut hour: i8 = hms.next()?.parse().ok()?; + let minute: i8 = hms.next()?.parse().ok()?; + let second: i8 = hms.next().unwrap_or("0").parse().ok()?; + match meridiem.to_ascii_uppercase().as_str() { + "PM" if hour < 12 => hour += 12, + "AM" if hour == 12 => hour = 0, + "AM" | "PM" => {} + _ => return None, + } + let dt = jiff::civil::date(year, month, day).at(hour, minute, second, 0); + dt.to_zoned(jiff::tz::TimeZone::UTC) + .ok() + .map(|z| z.timestamp()) +} + +fn parse_opencode_markdown(text: &str, anchor: jiff::Timestamp) -> Vec { + let anchor = opencode_created(text).unwrap_or(anchor); + let mut out: Vec = Vec::new(); + let mut role: Option<&'static str> = None; + let mut body = String::new(); + + let flush = |role: Option<&'static str>, body: &mut String, out: &mut Vec| { + let Some(role) = role else { + body.clear(); + return; + }; + let text = body.trim().to_string(); + body.clear(); + if text.is_empty() { + return; + } + let index = out.len(); + out.push(Message { + agent: "opencode".to_string(), + role: role.to_string(), + text, + created_at: anchor_series(anchor, index), + approximate: true, + }); + }; + + for line in text.lines() { + if let Some(next) = markdown_speaker(line) { + flush(role.take(), &mut body, &mut out); + role = Some(next); + continue; + } + if role.is_some() && line.trim_end() != "---" { + body.push_str(line); + body.push('\n'); + } + } + flush(role.take(), &mut body, &mut out); + out +} + +// --------------------------------------------------------------- scrollback + +/// Parses terminal scrollback pasted into a file. +/// +/// The lossiest path engram offers, and gated behind an explicit `--format` +/// for that reason. There is no structure here — the speaker is a glyph in +/// column zero (`❯` a prompt, `●` a reply, `⎿` a tool result, `✻` a spinner), +/// the text was hard-wrapped to the terminal's width with a two-space +/// continuation indent, and no timestamp survives anywhere. Wrapping cannot be +/// undone faithfully, so continuation lines are re-joined with a space and the +/// original line breaks are simply gone. +fn parse_claude_scrollback(text: &str, anchor: jiff::Timestamp) -> Vec { + let mut out: Vec = Vec::new(); + let mut role: Option<&'static str> = None; + let mut body = String::new(); + + let flush = |role: Option<&'static str>, body: &mut String, out: &mut Vec| { + let Some(role) = role else { + body.clear(); + return; + }; + let text = body.trim().to_string(); + body.clear(); + if text.is_empty() { + return; + } + let index = out.len(); + out.push(Message { + agent: "claude-code".to_string(), + role: role.to_string(), + text, + created_at: anchor_series(anchor, index), + approximate: true, + }); + }; + + for line in text.lines() { + let mut chars = line.chars(); + match chars.next() { + Some('❯') => { + flush(role.take(), &mut body, &mut out); + role = Some("user"); + body.push_str(chars.as_str().trim()); + } + Some('●') => { + flush(role.take(), &mut body, &mut out); + role = Some("assistant"); + body.push_str(chars.as_str().trim()); + } + // Tool results, spinners and status lines are chrome, not speech — + // the same payloads `transcript` excludes by default. + Some('⎿' | '✻' | '⏺') => { + flush(role.take(), &mut body, &mut out); + } + _ if role.is_some() => { + let t = line.trim(); + if !t.is_empty() { + if !body.is_empty() { + body.push(' '); + } + body.push_str(t); + } + } + _ => {} + } + } + flush(role.take(), &mut body, &mut out); + out +} + +/// The stable id of an imported message. +/// +/// Over the message's own content rather than its file, which is what makes +/// every dedupe requirement fall out of `INSERT OR IGNORE` instead of a +/// separate pass: re-importing an archive inserts nothing, thirteen +/// byte-identical copies of one file import once, and an archive holding two +/// concatenated copies of itself (the old `save-chat` append bug, still on +/// disk) yields one set of messages. Scope is included so the same archive +/// filed under two scopes really does land in both. +#[must_use] +pub fn import_id(scope: &str, m: &Message) -> String { + uuid::Uuid::new_v5( + &uuid::Uuid::NAMESPACE_OID, + format!( + "engram-import:{scope}:{}:{}:{}:{}", + m.agent, m.role, m.created_at, m.text + ) + .as_bytes(), + ) + .to_string() +} + +/// Reads and parses one export file. +/// +/// # Errors +/// +/// Propagates I/O failures, refuses a file above `max_bytes`, and returns +/// [`ImportError::Unrecognised`] when no format matches and none was forced. +pub fn read_file( + path: &Path, + forced: Option, + max_bytes: u64, +) -> Result<(Format, Vec), ImportError> { + let meta = std::fs::metadata(path).map_err(ImportError::Io)?; + if meta.len() > max_bytes { + return Err(ImportError::TooLarge { + bytes: meta.len(), + max_bytes, + }); + } + let text = std::fs::read_to_string(path).map_err(ImportError::Io)?; + let format = forced + .or_else(|| sniff(&text)) + .ok_or(ImportError::Unrecognised)?; + let anchor = file_anchor(&meta); + let messages = parse(&text, format, anchor)?; + Ok((format, messages)) +} + +/// The instant a timestamp-less export is anchored to: its own mtime. +/// +/// Not the wall clock. An archive's mtime is at least *about* when the +/// conversation happened, so ordering between files stays roughly right, while +/// `now()` would stack every historical import into the present and sort them +/// by the order they happened to be read. +fn file_anchor(meta: &std::fs::Metadata) -> jiff::Timestamp { + meta.modified() + .ok() + .and_then(|t| jiff::Timestamp::try_from(t).ok()) + .unwrap_or_else(jiff::Timestamp::now) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn anchor() -> jiff::Timestamp { + "2026-01-01T00:00:00Z".parse().expect("anchor") + } + + #[test] + fn unescape_reverses_exactly_three_characters() { + assert_eq!(unescape_texinfo("@@code @{a@} b"), "@code {a} b"); + // A left-to-right scan: the escaped `@@` is consumed before the brace + // rule can see the `{` behind it. + assert_eq!(unescape_texinfo("@@{"), "@{"); + // An `@` that is not an escape survives untouched. + assert_eq!(unescape_texinfo("user@host"), "user@host"); + } + + #[test] + fn heading_parses_both_dialects() { + let v2 = "@chapter Message by claude-code (user) at 2026-08-09T22:18:52.917Z"; + let v1 = "@section Message by codex (assistant) at 2026-07-26T00:04:16.394725123Z"; + assert_eq!( + parse_message_heading(v2), + Some(( + "claude-code".to_string(), + "user".to_string(), + "2026-08-09T22:18:52.917Z".to_string() + )) + ); + assert_eq!( + parse_message_heading(v1).map(|(a, r, _)| (a, r)), + Some(("codex".to_string(), "assistant".to_string())) + ); + // A document title is not a message. + assert_eq!( + parse_message_heading("@chapter Chat history for scope: engram"), + None + ); + } + + /// The third role survives. A corpus of 7,163 archived messages held 136 + /// `note` rows; folding them into `user` or `assistant` would rewrite what + /// the archive said to fit a two-variant type. + #[test] + fn texinfo_keeps_the_note_role() { + let doc = "\\input texinfo\n\ + @chapter Message by test-model (note) at 2026-07-11T08:08:35Z\n\ + Hello from model\n\ + @bye\n"; + let msgs = parse_texinfo(doc).expect("parses"); + assert_eq!(msgs.len(), 1); + assert_eq!(msgs[0].role, "note"); + assert_eq!(msgs[0].text, "Hello from model"); + assert!(!msgs[0].approximate); + } + + /// The old `save-chat` appended instead of rewriting, so one archive on + /// disk holds two copies of the same document. Identity is the content, so + /// the duplicate collapses at insert time. + #[test] + fn a_double_appended_archive_yields_duplicate_ids() { + // The real file interleaves a provenance comment and a title chapter + // between the two copies — that is what used to leak into the body. + let one = "@c Signed by: a-model on 2026-07-11T08:08:39Z\n\ + @chapter Chat history for scope: plan-test\n\ + @section Message by test-model (note) at 2026-07-11T08:08:35Z\n\ + Hello\n"; + let doc = format!("\\input texinfo\n{one}{one}@bye\n"); + let msgs = parse_texinfo(&doc).expect("parses"); + assert_eq!(msgs.len(), 2, "both copies are parsed"); + assert_eq!(msgs[0].text, "Hello", "no structural line leaked in"); + assert_eq!( + import_id("s", &msgs[0]), + import_id("s", &msgs[1]), + "and collapse to one row on INSERT OR IGNORE" + ); + } + + #[test] + fn markdown_ignores_headings_inside_a_reply() { + let doc = "# Project\n\n**Session ID:** ses_x\n\n---\n\n\ + ## User\n\nfix the build\n\n---\n\n\ + ## Assistant (Build · Model · 4.7s)\n\n\ + ## Root Cause Analysis\n\nthe flag moved\n\n\ + ## Next Move\n\nrename it\n\n---\n"; + let msgs = parse_opencode_markdown(doc, anchor()); + assert_eq!(msgs.len(), 2, "one user turn and one assistant turn"); + assert_eq!(msgs[0].role, "user"); + assert!(msgs[1].text.contains("Root Cause Analysis")); + assert!(msgs[1].text.contains("rename it")); + } + + /// Ordering is exact even though the absolute times are not. + #[test] + fn synthesised_timestamps_are_ordered_and_flagged() { + let doc = "**Session ID:** x\n## User\na\n## Assistant\nb\n## User\nc\n"; + let msgs = parse_opencode_markdown(doc, anchor()); + assert_eq!(msgs.len(), 3); + assert!(msgs.iter().all(|m| m.approximate)); + assert!(msgs[0].created_at < msgs[1].created_at); + assert!(msgs[1].created_at < msgs[2].created_at); + } + + #[test] + fn opencode_header_time_beats_the_file_anchor() { + let doc = "**Session ID:** x\n**Created:** 7/7/2026, 9:39:41 PM\n## User\na\n"; + let msgs = parse_opencode_markdown(doc, anchor()); + assert!( + msgs[0].created_at.starts_with("2026-07-07T21:39:41"), + "got {}", + msgs[0].created_at + ); + } + + #[test] + fn scrollback_uses_glyphs_and_drops_chrome() { + let doc = "❯ finish the feature\n because it is missing\n\ + ⎿ Listed directory\n\ + ● I'll start by loading\n the six skills\n\ + ✻ Baked for 5m 21s\n"; + let msgs = parse_claude_scrollback(doc, anchor()); + assert_eq!(msgs.len(), 2); + assert_eq!(msgs[0].role, "user"); + // Hard wrapping is re-joined; the original breaks are unrecoverable. + assert_eq!(msgs[0].text, "finish the feature because it is missing"); + assert_eq!(msgs[1].role, "assistant"); + assert!(!msgs[1].text.contains("Baked for")); + } + + #[test] + fn sniff_recognises_each_family_and_declines_the_rest() { + assert_eq!( + sniff("\\input texinfo\n@bye\n"), + Some(Format::EngramTexinfo) + ); + assert_eq!( + sniff("**Session ID:** x\n## User\nhi\n"), + Some(Format::OpencodeMarkdown) + ); + assert_eq!(sniff("❯ hello\n"), Some(Format::ClaudeScrollback)); + // The palette-editor page that shared a `chat/` directory with real + // archives: an extension check would have taken it, a content check + // does not. + assert_eq!(sniff("\nPalette Lab"), None); + } + + #[test] + fn a_bad_timestamp_is_an_error_not_a_substitution() { + let doc = "\\input texinfo\n@chapter Message by a (user) at not-a-time\nhi\n@bye\n"; + assert!(matches!( + parse_texinfo(doc), + Err(ImportError::BadTimestamp { .. }) + )); + } +} diff --git a/src/main.rs b/src/main.rs index d946f66..47d6194 100644 --- a/src/main.rs +++ b/src/main.rs @@ -11,6 +11,7 @@ mod error; mod facts; mod harness; mod http; +mod import; mod install; mod managed_file; mod mcp; @@ -678,6 +679,24 @@ fn run( ) } + Command::Import { + paths, + scope, + input_format, + recursive, + max_bytes, + dry_run, + } => handle_import( + &store, + mode, + &paths, + scope, + input_format, + recursive, + max_bytes, + dry_run, + ), + Command::Install { harness: requested, db_path, @@ -1553,6 +1572,203 @@ fn fail(err: AppError, mode: OutputMode) -> i32 { err.exit_code } +/// One export that produced messages. +#[derive(serde::Serialize)] +struct ImportedFile { + path: String, + format: &'static str, + scope: String, + messages: usize, + inserted: usize, + skipped_existing: usize, + /// True when the export carried no per-message times and they were + /// synthesised in order. Surfaced per file so a reader can tell an + /// approximate history from a recorded one without opening it. + approximate_times: bool, +} + +/// An export that produced nothing, and why. +#[derive(serde::Serialize)] +struct SkippedFile { + path: String, + reason: String, +} + +/// `engram import` payload. +#[derive(serde::Serialize)] +struct ImportResult { + dry_run: bool, + files: Vec, + /// Never empty on a real corpus: a `chat/` directory accumulates whatever + /// was dropped in it. Reporting each skip with its reason is the + /// difference between "engram found nothing" and "engram found nine files + /// it could not read". + skipped: Vec, + scopes: Vec, + inserted: usize, + skipped_existing: usize, + filtered: transcript::FilterStats, + redactions: transcript::redact::Redactions, +} + +/// Collects the files an import should consider. +/// +/// Walks with `std::fs` and therefore ignores `.gitignore` entirely, which is +/// the only thing that makes this command useful: engram itself adds `chat/` +/// to a project's `.gitignore` the first time `save-chat` runs, so every +/// archive it has ever written is ignored by definition. A walker that +/// honoured ignore rules — as `fd`, `rg` and `git ls-files` all do by default +/// — would find precisely zero of them and report success. +fn collect_exports(paths: &[std::path::PathBuf], recursive: bool) -> Vec { + let mut out = Vec::new(); + let mut stack: Vec = paths.to_vec(); + while let Some(p) = stack.pop() { + if p.is_file() { + out.push(p); + continue; + } + let Ok(entries) = std::fs::read_dir(&p) else { + continue; + }; + for entry in entries.flatten() { + let path = entry.path(); + if path.is_dir() { + if recursive { + stack.push(path); + } + } else { + out.push(path); + } + } + } + out.sort(); + out +} + +/// Imports exported transcripts into the store. +#[allow(clippy::too_many_arguments, clippy::too_many_lines)] +fn handle_import( + store: &std::sync::Mutex, + mode: OutputMode, + paths: &[std::path::PathBuf], + scope: Option, + format: Option, + recursive: bool, + max_bytes: u64, + dry_run: bool, +) -> i32 { + let files = collect_exports(paths, recursive); + let mut result = ImportResult { + dry_run, + files: Vec::new(), + skipped: Vec::new(), + scopes: Vec::new(), + inserted: 0, + skipped_existing: 0, + filtered: transcript::FilterStats::default(), + redactions: transcript::redact::Redactions::default(), + }; + + for path in files { + let display = path.display().to_string(); + let (fmt, messages) = match import::read_file(&path, format, max_bytes) { + Ok(v) => v, + Err(e) => { + result.skipped.push(SkippedFile { + path: display, + reason: e.to_string(), + }); + continue; + } + }; + if messages.is_empty() { + result.skipped.push(SkippedFile { + path: display, + reason: "recognised the format but found no messages".to_string(), + }); + continue; + } + + // Scope per file, resolved from the directory the export sits in, so + // one run over many projects files each into its own scope. + // + // Canonicalized first: `engram import ./chat` resolves its parent to + // `.`, whose basename is empty, and the scope silently became + // `default` — one relative path away from filing an entire corpus + // under the wrong name. + let base = path.parent().unwrap_or(&path); + let base = std::fs::canonicalize(base).unwrap_or_else(|_| base.to_path_buf()); + let resolved = rules::resolve_scope_in(scope.as_deref(), &base); + + let approximate = messages.iter().any(|m| m.approximate); + let mut rows: Vec = Vec::with_capacity(messages.len()); + for m in &messages { + // The same normalization and redaction the live readers apply. An + // archive is not more trustworthy for being old — these files hold + // whatever was pasted into the session. + let Some(text) = transcript::normalize_text( + &m.text, + transcript::DEFAULT_MAX_CHARS_PER_TURN, + &mut result.filtered, + ) else { + continue; + }; + let text = transcript::redact::scrub(&text, &mut result.redactions); + let scrubbed = import::Message { text, ..m.clone() }; + rows.push(store::IngestTurn { + id: import::import_id(&resolved.name, &scrubbed), + agent: scrubbed.agent.clone(), + role: scrubbed.role.clone(), + content: scrubbed.text.clone(), + created_at: scrubbed.created_at.clone(), + }); + } + + let report = if dry_run { + store::IngestReport { + inserted: rows.len(), + skipped_existing: 0, + } + } else { + let mut guard = store.lock().expect("store lock poisoned"); + match guard.ingest_turns(&resolved.name, &rows) { + Ok(r) => r, + Err(e) => return fail(AppError::from(e), mode), + } + }; + + result.inserted += report.inserted; + result.skipped_existing += report.skipped_existing; + if !result.scopes.contains(&resolved.name) { + result.scopes.push(resolved.name.clone()); + } + result.files.push(ImportedFile { + path: display, + format: fmt.as_str(), + scope: resolved.name, + messages: rows.len(), + inserted: report.inserted, + skipped_existing: report.skipped_existing, + approximate_times: approximate, + }); + } + + result.scopes.sort(); + let label = if dry_run { + "engram import --dry-run" + } else { + "engram import" + }; + let response = Response::new(label, result); + let response = if dry_run { + response.with_dry_run() + } else { + response + }; + emit_ok(response, mode); + 0 +} + // Rust guideline compliant 2026-05-18 fn emit_ok(resp: Response, mode: OutputMode) { diff --git a/tests/cli.rs b/tests/cli.rs index 9d40d7f..36689a9 100644 --- a/tests/cli.rs +++ b/tests/cli.rs @@ -3617,3 +3617,192 @@ fn ingest_cwd_sets_the_scope_and_explicit_scope_still_wins() { assert_eq!(data["scope"], "pinned"); assert_eq!(data["scope_origin"], "explicit"); } + +/// An `engram import` invocation against a project directory. +fn import(db: &Path, project: &Path, args: &[&str]) -> Command { + let mut cmd = engram(db); + cmd.current_dir(project); + cmd.arg("import"); + cmd.args(args); + cmd +} + +/// Writes a small engram archive into `/chat/`. +fn plant_archive(project: &Path, name: &str, body: &str) -> std::path::PathBuf { + let dir = project.join("chat"); + std::fs::create_dir_all(&dir).expect("create chat dir"); + let path = dir.join(name); + std::fs::write(&path, body).expect("write archive"); + path +} + +const ARCHIVE: &str = "\\input texinfo @c -*-texinfo-*-\n\ + @documentencoding UTF-8\n\ + @settitle Chat history for scope: whatever\n\n\ + @c Generated by `engram save-chat`. Edits are overwritten.\n\ + @node Top\n\ + @top Chat history for scope: whatever\n\n\ + @chapter Message by claude-code (user) at 2026-07-26T00:04:16.394Z\n\ + please fix the flag\n\n\ + @chapter Message by claude-code (assistant) at 2026-07-26T00:05:38.892Z\n\ + Renamed it and re-ran the gate.\n\n\ + @chapter Message by test-model (note) at 2026-07-26T00:06:00.000Z\n\ + archived for later\n\ + @bye\n"; + +/// Round-trips what `save-chat` writes, which engram could not read back. +/// +/// The scope comes from the directory the archive sits in, not from the +/// `@settitle` line — recorded scopes in the real corpus are inconsistent +/// (bare names, paths, absolute paths, branch names), so the containing +/// project is the reliable signal. +#[test] +fn import_reads_an_engram_archive_and_is_idempotent() { + let tmp = TempDir::new().expect("tempdir"); + let db = tmp.path().join("test.db"); + let project = pinned_project(&tmp); + plant_archive(&project, "2026-07-26T00-00-00Z.texi", ARCHIVE); + + let assert = import(&db, &project, &["chat"]).assert().success(); + let data = parse_single_line_json(&assert.get_output().stdout)["data"].clone(); + assert_eq!(data["inserted"], 3); + assert_eq!( + data["scopes"][0], "project", + "scope is the containing project" + ); + assert_eq!(data["files"][0]["format"], "engram-texinfo"); + assert_eq!(data["files"][0]["approximate_times"], false); + + // The third role survives rather than being coerced. + let assert = engram(&db) + .args(["recall", "--scope", "project", "--limit", "10"]) + .assert() + .success(); + let roles: Vec = parse_single_line_json(&assert.get_output().stdout)["data"] + .as_array() + .expect("memories") + .iter() + .map(|m| m["role"].as_str().unwrap_or_default().to_string()) + .collect(); + assert!(roles.contains(&"note".to_string()), "roles: {roles:?}"); + assert!(roles.contains(&"user".to_string())); + assert!(roles.contains(&"assistant".to_string())); + + // Identity is the content, so a second run writes nothing. + let assert = import(&db, &project, &["chat"]).assert().success(); + let data = parse_single_line_json(&assert.get_output().stdout)["data"].clone(); + assert_eq!(data["inserted"], 0); + assert_eq!(data["skipped_existing"], 3); +} + +/// A file that is not a transcript is skipped with a reason, not an error. +/// +/// A `chat/` directory accumulates whatever was dropped in it — the real one +/// held a standalone HTML palette editor alongside real archives. Sniffing +/// content rather than extension is what keeps that out, and reporting the +/// skip is what distinguishes "found nothing" from "could not read nine +/// files". +#[test] +fn import_skips_what_it_cannot_recognise_and_says_so() { + let tmp = TempDir::new().expect("tempdir"); + let db = tmp.path().join("test.db"); + let project = pinned_project(&tmp); + plant_archive(&project, "real.texi", ARCHIVE); + plant_archive( + &project, + "palette.html", + "\nPalette Lab\n\n", + ); + + let assert = import(&db, &project, &["chat"]).assert().success(); + let data = parse_single_line_json(&assert.get_output().stdout)["data"].clone(); + + assert_eq!(data["files"].as_array().expect("files").len(), 1); + let skipped = data["skipped"].as_array().expect("skipped"); + assert_eq!(skipped.len(), 1); + assert!( + skipped[0]["path"] + .as_str() + .expect("path") + .ends_with(".html"), + "{skipped:?}" + ); + assert!( + skipped[0]["reason"] + .as_str() + .expect("reason") + .contains("no known export format"), + "{skipped:?}" + ); +} + +/// One run over several projects files each into its own scope. +#[test] +fn import_files_each_project_into_its_own_scope() { + let tmp = TempDir::new().expect("tempdir"); + let db = tmp.path().join("test.db"); + + for name in ["alpha", "beta"] { + let project = tmp.path().join(name); + std::fs::create_dir_all(&project).expect("create project"); + std::fs::write(project.join(".git"), "gitdir: elsewhere\n").expect("pin git root"); + plant_archive(&project, "a.texi", ARCHIVE); + } + + let here = pinned_project(&tmp); + let alpha = tmp.path().join("alpha").to_string_lossy().into_owned(); + let beta = tmp.path().join("beta").to_string_lossy().into_owned(); + let assert = import(&db, &here, &[&alpha, &beta, "--recursive"]) + .assert() + .success(); + let data = parse_single_line_json(&assert.get_output().stdout)["data"].clone(); + + let mut scopes: Vec = data["scopes"] + .as_array() + .expect("scopes") + .iter() + .map(|s| s.as_str().unwrap_or_default().to_string()) + .collect(); + scopes.sort(); + assert_eq!(scopes, vec!["alpha".to_string(), "beta".to_string()]); +} + +/// Timestamp-less exports are ordered, flagged, and still importable. +#[test] +fn import_synthesises_ordered_times_for_a_markdown_export() { + let tmp = TempDir::new().expect("tempdir"); + let db = tmp.path().join("test.db"); + let project = pinned_project(&tmp); + plant_archive( + &project, + "session.md", + "# Project\n\n**Session ID:** ses_x\n**Created:** 7/7/2026, 9:39:41 PM\n\n---\n\n\ + ## User\n\nfix the build\n\n---\n\n\ + ## Assistant (Build · Model · 4.7s)\n\n\ + ## Root Cause Analysis\n\nthe flag moved\n\n---\n", + ); + + let assert = import(&db, &project, &["chat"]).assert().success(); + let data = parse_single_line_json(&assert.get_output().stdout)["data"].clone(); + assert_eq!(data["files"][0]["format"], "opencode-markdown"); + assert_eq!(data["files"][0]["approximate_times"], true); + assert_eq!( + data["inserted"], 2, + "a heading inside a reply is not a turn" + ); + + // Order is exact even though the absolute times are not. + let assert = engram(&db) + .args(["recall", "--scope", "project"]) + .assert() + .success(); + let mems = parse_single_line_json(&assert.get_output().stdout)["data"].clone(); + let times: Vec<&str> = mems + .as_array() + .expect("memories") + .iter() + .map(|m| m["created_at"].as_str().unwrap_or_default()) + .collect(); + assert!(times[0] < times[1], "{times:?}"); + assert!(times[0].starts_with("2026-07-07T21:39:41"), "{times:?}"); +}