From 8ce264c36235c32d65ba32ca248a4ff3ab1e0ba0 Mon Sep 17 00:00:00 2001 From: UnbreakableMJ Date: Tue, 25 Aug 2026 23:14:14 +0300 Subject: [PATCH] Add engram import, for transcripts a harness exported MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ingest` reads the session file a harness writes for itself, which reaches a harness engram has a reader for and nothing else. That left two gaps. A harness may keep its history in a form engram cannot parse — protocol buffers, editor workspace state — while still offering its own export command. And an archive is not a live session: `save-chat` wrote `.texi` files engram had no way to read back, so engram's own output was the one format it could not ingest. `import` is deliberately not a `ReaderKind`. That enum is built around "one path per installed harness", discovered from a HarnessSpec and a working directory, and a file somebody hands you fits none of it. It feeds the shared pipeline directly instead: normalize_text, redact::scrub, ingest_turns. Identity is the message's content, not its file — a v5 uuid over scope, agent, role, timestamp and text. Every dedupe requirement then falls out of INSERT OR IGNORE with no separate pass: re-importing inserts 0, thirteen byte-identical copies of one archive import once, and the archive holding two concatenated copies of itself — residue of an earlier save-chat that appended rather than rewrote — collapses to one set of messages. Formats are sniffed by content, never by extension: a real chat/ directory held a standalone HTML palette editor next to genuine archives, and several source trees merely named `chat` are skipped the same way. Everything unrecognised is reported with its reason, which is the difference between "found nothing" and "could not read nine files". Three findings worth recording because each cost a debugging pass: A structural line is never message content — escaping doubles every literal `@`, so a single leading `@` is markup. Without that filter the trailing `@c Signed by:` and `@chapter Chat history…` of one document became the tail of the previous message's body, which is exactly why the double-appended archive did not deduplicate at first: the last message of each copy differed by the next copy's header. `--input-format`, not `--format`: the global output-format flag already owns that name, and a subcommand reusing it makes clap panic on the duplicate argument id rather than shadow it. The base directory is canonicalized before scope resolution. `import ./chat` resolves its parent to `.`, whose basename is empty, and the scope silently became `default` — one relative path away from filing a corpus under the wrong name. Missing timestamps are synthesised in order from a file-level anchor and flagged per file, never taken from the clock: stamping now() would collapse a conversation into one instant and destroy reading order, the same failure `transcript` refuses a wall-clock fallback to avoid. Verified against the real corpus: 100 files, 9,907 messages, 15 scopes. Gates: fmt, clippy -D warnings, 278 tests, REUSE 3.3, makeinfo clean. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_016i16R4GhdSffsboRYq97Fs --- AGENTS.md | 15 ++ CHANGELOG.md | 8 + doc/engram.texi | 65 +++++ src/cli.rs | 34 +++ src/import.rs | 684 ++++++++++++++++++++++++++++++++++++++++++++++++ src/main.rs | 216 +++++++++++++++ tests/cli.rs | 189 +++++++++++++ 7 files changed, 1211 insertions(+) create mode 100644 src/import.rs diff --git a/AGENTS.md b/AGENTS.md index cbb1dcc..c4e6ce7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -267,6 +267,21 @@ Two readers exist: `claude_code` and `codex`. Adding a third means adding a `Rea - **Redaction** (`redact.rs`) replaces credential-shaped substrings before storage and counts them per kind in the envelope. Best-effort, not a guarantee — it catches machine-issued token shapes, not a password typed in prose. The real defense is the default filtering above. `harness::home_dir()` reads `$HOME` directly rather than via the `dirs` crate: a **testability decision**, since every harness path derives from it and a test that sets `HOME` to a tempdir is then hermetic by construction. Do not turn it into a dependency. - **Fixtures are synthetic**, never copied sessions (`tests/fixtures/transcripts/README.md` explains why): a real transcript holds whatever the user pasted. +## Importing exported transcripts (`engram import`) + +`src/import.rs`. `ingest` reads the session file a harness writes for itself, which covers a harness engram has a reader for and nothing else. `import` reads what was *exported* — so it reaches a harness whose store engram cannot parse (protobuf, editor workspace state) but which offers its own export command, and it reads back engram's own `save-chat` archives, which were previously the one format engram could write and not read. + +- **Deliberately not a `ReaderKind`.** That enum is built around "one path per installed harness", discovered from a `HarnessSpec` plus a working directory. A file somebody hands you fits none of that, so `import` feeds the shared pipeline directly (`normalize_text` → `redact::scrub` → `Store::ingest_turns`) without touching `TranscriptSupport`. +- **Identity is the content, not the file.** `import_id` is a v5 uuid over `(scope, agent, role, created_at, text)`, so every dedupe requirement falls out of `INSERT OR IGNORE`: re-importing inserts 0, thirteen byte-identical copies of one file import once, and the archive holding two concatenated copies of itself (residue of the old appending `save-chat`) collapses to one set. No separate dedupe pass exists, deliberately. +- **Three formats, sniffed by content not extension.** engram Texinfo (both dialects — current `@chapter`, legacy `@section` — one heading grammar covers all 7,163 messages in the real corpus), Opencode Markdown, and Claude Code scrollback. A real `chat/` dir held a standalone HTML palette editor; extension-matching would have ingested it. Source directories merely *named* `chat` (Kotlin, TypeScript, Rust) are skipped the same way. Unrecognised files are reported in `skipped` **with a reason** — the difference between "found nothing" and "could not read nine files". +- **A structural line is never message content.** Escaping doubles every literal `@`, so any line starting with a single `@` is markup. Without that filter the trailing `@c Signed by:` / `@chapter Chat history…` of one document became the tail of the previous message's body — which is precisely why the double-appended archive failed to deduplicate at first: the last message of each copy differed by the *next* copy's header. +- **`--input-format`, not `--format`.** `--format` is the global output-format flag; a subcommand reusing the name makes clap panic on the duplicate argument id (`Mismatch between definition and access`), not shadow it. +- **Canonicalize before resolving scope.** `import ./chat` resolves its parent to `.`, whose basename is empty, and the scope silently became `default` — one relative path from filing a whole corpus under the wrong name. +- **Roles are not binary.** The corpus held `assistant` 6395, `user` 632, **`note` 136**. The third is carried through, not coerced. +- **Missing timestamps are synthesised in order, never from the clock.** Markdown and scrollback record no per-message time. Each message is offset one second from a file-level anchor (the export's recorded session time, else the file's mtime); `approximate_times` is reported per file. Stamping `now()` would collapse a conversation into one instant and destroy reading order — the same failure `transcript` refuses a wall-clock fallback to avoid. +- **The walker must not honour `.gitignore`.** engram itself adds `chat/` to a project's `.gitignore` on first `save-chat`, so every archive it has written is ignored by definition; `fd`, `rg` and `git ls-files` all find zero of them by default. `std::fs` ignores ignore-rules, which is the only reason this command works. +- **Known unhandled variants**, both real and both reported as skips: older Claude Code scrollback with no `❯` prompt marker (`--input-format claude-scrollback` recovers the `●` assistant turns only), and Goose terminal output (`> ` marks the user). + ## Harness command delivery (`engram install`) `src/install.rs` + `plugins/engram/`. Engram was already an MCP server in every harness on a typical machine; what was missing was a *command surface*. diff --git a/CHANGELOG.md b/CHANGELOG.md index ae05a13..98cd2ac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,6 +47,14 @@ follows [Keep a Changelog](https://keepachangelog.com/); versions follow ### Added +- **`engram import`** — reads chat transcripts that were exported to files, for + the harnesses `ingest` has no reader for, and reads back `save-chat`'s own + `.texi` archives. Three formats, detected by content: engram Texinfo (both + dialects), Opencode Markdown, and Claude Code scrollback. Each file is filed + into the scope of the project containing it; identity is a v5 uuid over the + message's content, so re-importing inserts nothing and duplicate files + collapse. `--input-format` forces a parser, `--recursive` descends. + - **VS Code** and **Cursor** are supported harnesses (the tenth and eleventh). VS Code gets `~/.config/Code/User/prompts/engram-.prompt.md`; Cursor gets `~/.cursor/skills/engram-/SKILL.md`. VS Code's `mcp.json` keys its diff --git a/doc/engram.texi b/doc/engram.texi index 45e7227..596eae8 100644 --- a/doc/engram.texi +++ b/doc/engram.texi @@ -827,6 +827,71 @@ Transcripts can be very large --- one rollout on a developer machine was @code{--max-bytes} (64@tie{}MiB by default) refuses an oversized transcript with an error naming the override rather than attempting it. +@section engram import + +@cindex import command +@cindex exported transcripts +@cindex chat archives + +@example +engram import @var{path}@dots{} [--scope @var{id}] [--input-format @var{fmt}] + [--recursive] [--max-bytes @var{n}] [--dry-run] +@end example + +Reads chat transcripts that were @emph{exported} to files, rather than the +session file a harness writes for itself. @command{engram ingest} covers a +harness engram has a reader for; this covers everything else. Two cases +motivate it. A harness may keep its history in a form engram cannot parse +--- protocol buffers, an editor's workspace state --- while still offering +its own export command. And an archive is not a live session at all: +@command{engram save-chat} could write a @file{.texi} that engram had no way +to read back, so engram's own output was the one format it could not ingest. + +Three formats are recognised, by @emph{content} rather than by extension: +archives written by @command{save-chat} (both dialects, the current +@code{@@chapter} form and the older @code{@@section} one), Opencode's +Markdown export, and raw terminal scrollback. A file that announces none of +them is skipped with a reason rather than guessed at --- a real @file{chat/} +directory held a standalone HTML palette editor alongside genuine archives, +and source files in directories that merely happen to be named @file{chat} +are skipped the same way. @code{--input-format} forces a parser for a file +that cannot be detected. It is spelled that way because @code{--format} is +the global output-format option. + +@cindex scope, on import +Each file is filed into the scope of the project @emph{containing} it, so one +run over many directories fills many scopes correctly. The scope recorded +inside an archive is deliberately ignored: in practice those strings are +inconsistent --- bare names, relative paths, absolute paths and branch names +all appear --- while the containing project is unambiguous. + +@cindex idempotence, of import +A message's identity is a version@tie{}5 UUID over its scope, agent, role, +timestamp and text, so every deduplication requirement falls out of +@code{INSERT OR IGNORE} rather than a separate pass. Re-importing an archive +inserts nothing. A file copied into four projects imports once per scope +instead of four times. An archive holding two concatenated copies of itself +--- the residue of an earlier @command{save-chat} that appended rather than +rewrote --- collapses back to one set of messages. + +@cindex roles, third +Roles are not binary. A real corpus of 7,163 archived messages held 6,395 +@code{assistant}, 632 @code{user} and 136 @code{note}; the third is carried +through rather than coerced into one of the other two. + +@cindex timestamps, synthesised +The Markdown and scrollback formats record no per-message time. Rather than +refuse them or stamp them from the clock --- which would collapse a whole +conversation into one instant and destroy reading order, the failure +@command{engram ingest} refuses a wall-clock fallback to avoid --- each +message is offset one second from a file-level anchor: the session time the +export records, else the file's own modification time. Absolute values are +approximate and the response says so per file, in +@code{approximate_times}; the ordering is exact. + +Text is normalized and redacted on the way in, exactly as a live transcript +is. An archive is not more trustworthy for being old. + @section engram install @cindex install command diff --git a/src/cli.rs b/src/cli.rs index e2965a2..1562b22 100644 --- a/src/cli.rs +++ b/src/cli.rs @@ -347,6 +347,40 @@ EXAMPLES: #[arg(long)] dry_run: bool, }, + /// Import chat transcripts that were exported to files. + /// + /// Reads what a harness's own export command wrote, and what `save-chat` + /// archived, for the harnesses engram has no reader for. + Import { + /// Files or directories to import. A directory is scanned for exports. + #[arg(value_name = "PATH", required = true)] + paths: Vec, + + /// Scope to store into. Resolves from each file's own project when + /// omitted, so one run can fill many scopes correctly. + #[arg(long)] + scope: Option, + + /// Force a parser instead of detecting one from the file's content. + /// + /// Deliberately not `--format`: that is the global output-format flag, + /// and giving a subcommand the same name makes clap panic on the + /// duplicate argument id rather than shadow it. + #[arg(long, value_enum)] + input_format: Option, + + /// Descend into subdirectories. + #[arg(long)] + recursive: bool, + + /// Refuse an export larger than this many bytes. + #[arg(long, default_value_t = crate::transcript::DEFAULT_MAX_BYTES)] + max_bytes: u64, + + /// Parse and report what would be stored, without writing. + #[arg(long)] + dry_run: bool, + }, /// Write engram's slash commands into the harnesses on this machine. /// /// Engram is usually already registered as an MCP server everywhere; what diff --git a/src/import.rs b/src/import.rs new file mode 100644 index 0000000..b55771c --- /dev/null +++ b/src/import.rs @@ -0,0 +1,684 @@ +// SPDX-FileCopyrightText: 2026 Mohamed Hammad +// SPDX-License-Identifier: GPL-3.0-or-later + +//! Importing chat transcripts that were *exported* rather than read live. +//! +//! [`crate::transcript`] reads the session file a harness writes for itself. +//! That covers a harness engram has a reader for, and nothing else — which +//! left two gaps. A harness may store its history somewhere engram cannot +//! parse (protocol buffers, an editor's workspace state) while still offering +//! its own export command. And an archive written months ago is not a live +//! session at all: `save-chat` could write a `.texi` that engram had no way to +//! read back, so engram's own output was the one format it could not ingest. +//! +//! An export file is the common denominator, so this module takes one and +//! turns it into the same rows `transcript::capture` produces. It is +//! deliberately **not** a [`crate::harness::ReaderKind`]: that enum is built +//! around "one path per installed harness", discovered from a `HarnessSpec` +//! and a working directory, and a file somebody hands you fits none of that. +//! +//! Three properties are worth stating because each was learned from the +//! corpus rather than guessed: +//! +//! * **Identity is the content, not the file.** A message's id is a v5 uuid +//! over `(scope, agent, role, created_at, text)`, so re-importing the same +//! archive inserts nothing, a file copied to four projects imports once per +//! scope rather than four times per scope, and the old `save-chat` append +//! bug — which left one on-disk archive holding two concatenated copies of +//! the same document — collapses back to one set of messages. No separate +//! dedupe pass is needed; `INSERT OR IGNORE` does it. +//! * **Roles are not binary.** A real corpus of 7,163 archived messages held +//! `assistant` 6,395, `user` 632 and **`note` 136**. Coercing that third +//! role into one of the other two would rewrite history to fit a type. +//! * **A missing timestamp is synthesised in order, never invented from the +//! clock.** See [`anchor_series`]. + +use std::path::Path; + +/// An export format engram can parse. +#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize, clap::ValueEnum)] +#[serde(rename_all = "kebab-case")] +#[clap(rename_all = "kebab-case")] +pub enum Format { + /// A `.texi` written by `engram save-chat`, either dialect. + EngramTexinfo, + /// Opencode's (and Kilo's) Markdown export. + OpencodeMarkdown, + /// Raw Claude Code terminal scrollback, pasted into a file. + ClaudeScrollback, +} + +impl Format { + /// The `--format` value and the name reported back. + #[must_use] + pub fn as_str(self) -> &'static str { + match self { + Format::EngramTexinfo => "engram-texinfo", + Format::OpencodeMarkdown => "opencode-markdown", + Format::ClaudeScrollback => "claude-scrollback", + } + } +} + +/// One message recovered from an export. +#[derive(Debug, Clone, serde::Serialize)] +pub struct Message { + pub agent: String, + /// `user`, `assistant` or `note` — whatever the archive recorded. + pub role: String, + pub text: String, + pub created_at: String, + /// True when [`anchor_series`] supplied the timestamp. + pub approximate: bool, +} + +/// Why an import could not proceed. +#[derive(Debug)] +pub enum ImportError { + Io(std::io::Error), + TooLarge { + bytes: u64, + max_bytes: u64, + }, + /// Nothing in the file looked like a transcript engram knows. + Unrecognised, + /// A recognised format whose timestamp did not parse. As in + /// [`crate::transcript`], this is an error rather than a substitution: + /// recall orders by `created_at`, so a wrong value corrupts reading order + /// invisibly. + BadTimestamp { + value: String, + }, +} + +impl std::fmt::Display for ImportError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + ImportError::Io(e) => write!(f, "{e}"), + ImportError::TooLarge { bytes, max_bytes } => { + write!( + f, + "export is {bytes} bytes, above the {max_bytes}-byte ceiling" + ) + } + ImportError::Unrecognised => { + write!(f, "no known export format matched this file") + } + ImportError::BadTimestamp { value } => { + write!(f, "unparseable timestamp: {value}") + } + } + } +} + +/// Identifies the format of an export by looking at its content. +/// +/// Content, not extension, on purpose: the corpus contained two `.html` files +/// in a `chat/` directory that were a standalone palette editor rather than a +/// conversation, and a future `.html` genuinely could be a web export. A file +/// that does not announce itself is skipped rather than guessed at. +#[must_use] +pub fn sniff(text: &str) -> Option { + if text.contains("\\input texinfo") { + return Some(Format::EngramTexinfo); + } + // Opencode's export leads with a `**Session ID:**` block and marks every + // speaker with a level-two heading. + if text.contains("**Session ID:**") + || text + .lines() + .any(|l| l.starts_with("## Assistant (") || l.trim_end() == "## Assistant") + { + return Some(Format::OpencodeMarkdown); + } + // Scrollback has no headings at all — the speaker is a glyph in column 0. + if text.lines().any(|l| l.starts_with('❯')) { + return Some(Format::ClaudeScrollback); + } + None +} + +/// Parses an export whose format is already known. +/// +/// `anchor` is the fallback instant for formats that do not record per-message +/// times; see [`anchor_series`]. +/// +/// # Errors +/// +/// Returns [`ImportError::BadTimestamp`] when a recorded timestamp does not +/// parse. +pub fn parse( + text: &str, + format: Format, + anchor: jiff::Timestamp, +) -> Result, ImportError> { + match format { + Format::EngramTexinfo => parse_texinfo(text), + Format::OpencodeMarkdown => Ok(parse_opencode_markdown(text, anchor)), + Format::ClaudeScrollback => Ok(parse_claude_scrollback(text, anchor)), + } +} + +/// Assigns the `index`-th message of a timestamp-less export its instant. +/// +/// One second per message from a file-level anchor. The absolute values are +/// approximate and say so (`Message::approximate`), but the *order* — the only +/// thing `recall` actually depends on — is exact. The alternative considered +/// and rejected was stamping every message with the same instant, which would +/// collapse a whole conversation into one moment and destroy reading order, +/// the precise failure [`crate::transcript`] refuses a wall-clock fallback to +/// avoid. +#[must_use] +pub fn anchor_series(anchor: jiff::Timestamp, index: usize) -> String { + let shifted = anchor + .checked_add(jiff::Span::new().seconds(i64::try_from(index).unwrap_or(i64::MAX))) + .unwrap_or(anchor); + // `jiff::Timestamp`'s Display is RFC 3339 with a `Z` suffix, which is + // exactly what `normalize_timestamp` produces for the recorded case. + shifted.to_string() +} + +// ------------------------------------------------------------------ texinfo + +/// Reverses [`crate::archive::escape_texinfo`]. +/// +/// That function maps exactly three characters — `@`→`@@`, `{`→`@{`, `}`→`@}` +/// — and nothing else, so this is a three-way reverse and not a general +/// Texinfo decoder. Scanning left to right matters: a naive +/// `replace("@@", "@")` run before the brace rules would turn an escaped +/// `@@{` into `@{` and then into a brace that was never there. +#[must_use] +pub fn unescape_texinfo(s: &str) -> String { + let mut out = String::with_capacity(s.len()); + let mut chars = s.chars(); + while let Some(c) = chars.next() { + if c == '@' { + match chars.next() { + Some(n @ ('@' | '{' | '}')) => out.push(n), + Some(other) => { + out.push('@'); + out.push(other); + } + None => out.push('@'), + } + } else { + out.push(c); + } + } + out +} + +/// Splits a `Message by () at ` heading. +/// +/// Parsed from the right, because an agent name is the only free-form part: +/// the timestamp is the last token, `(role)` the token before it. The two +/// archive dialects differ only in heading level — the current writer emits +/// `@chapter` (the document already has an `@top`), the older one `@section` +/// under a title chapter — so both are accepted and the level carries no +/// meaning beyond that. +fn parse_message_heading(line: &str) -> Option<(String, String, String)> { + let rest = line + .strip_prefix("@chapter Message by ") + .or_else(|| line.strip_prefix("@section Message by "))?; + let (left, ts) = rest.rsplit_once(" at ")?; + let left = left.trim_end(); + let open = left.rfind(" (")?; + let role = left[open + 2..].strip_suffix(')')?; + let agent = &left[..open]; + if agent.is_empty() || role.is_empty() || ts.trim().is_empty() { + return None; + } + Some((agent.to_string(), role.to_string(), ts.trim().to_string())) +} + +/// True for a Texinfo command line, which is never message content. +/// +/// `@@` is an escaped literal `@` and therefore *is* content; a single `@` +/// followed by anything else is markup engram wrote. +fn is_structural(line: &str) -> bool { + let mut chars = line.chars(); + chars.next() == Some('@') && chars.next() != Some('@') +} + +fn parse_texinfo(text: &str) -> Result, ImportError> { + let mut out: Vec = Vec::new(); + let mut current: Option<(String, String, String)> = None; + let mut body = String::new(); + + let flush = |head: Option<(String, String, String)>, + body: &mut String, + out: &mut Vec| + -> Result<(), ImportError> { + let Some((agent, role, ts)) = head else { + body.clear(); + return Ok(()); + }; + let created_at = crate::transcript::normalize_timestamp(&ts) + .map_err(|value| ImportError::BadTimestamp { value })?; + let text = unescape_texinfo(body.trim_end_matches('\n')); + body.clear(); + if text.trim().is_empty() { + return Ok(()); + } + out.push(Message { + agent: unescape_texinfo(&agent), + role, + text, + created_at, + approximate: false, + }); + Ok(()) + }; + + for line in text.lines() { + if let Some(head) = parse_message_heading(line) { + flush(current.take(), &mut body, &mut out)?; + current = Some(head); + continue; + } + if line.trim_end() == "@bye" { + flush(current.take(), &mut body, &mut out)?; + continue; + } + // A structural line can never be message content: escaping doubles + // every literal `@`, so content always arrives as `@@…`. Without this + // the trailing `@c Signed by:` and `@chapter Chat history…` lines of + // one document became the tail of the previous message's body — which + // is exactly how a file holding two concatenated copies of itself + // failed to deduplicate: the last message of each copy differed by the + // header of the next one. + if is_structural(line) { + continue; + } + if current.is_some() { + body.push_str(line); + body.push('\n'); + } + } + flush(current.take(), &mut body, &mut out)?; + Ok(out) +} + +// ----------------------------------------------------------------- markdown + +/// True for a line that opens a message in Opencode's export. +/// +/// Anchored on the two literal speakers rather than on `^## `, because an +/// assistant's own reply routinely contains level-two headings — the observed +/// export had `## Root Cause Analysis`, `## Objective`, `## Next Move` inside +/// message bodies. Splitting on every `## ` shattered one 294 KB conversation +/// of 100 messages into several hundred fragments. +fn markdown_speaker(line: &str) -> Option<&'static str> { + let t = line.trim_end(); + if t == "## User" || t.starts_with("## User (") { + return Some("user"); + } + if t == "## Assistant" || t.starts_with("## Assistant (") { + return Some("assistant"); + } + None +} + +/// Reads `**Created:** 7/7/2026, 9:39:41 PM` out of the export's header. +/// +/// US-locale, not RFC 3339 — the exporter formats for a reader, not for a +/// parser. Returns `None` rather than guessing when the shape does not match, +/// leaving the caller's file-level anchor in place. +fn opencode_created(text: &str) -> Option { + let line = text + .lines() + .take(40) + .find_map(|l| l.trim().strip_prefix("**Created:**"))?; + let (date, time) = line.trim().split_once(", ")?; + let mut parts = date.split('/'); + let month: i8 = parts.next()?.trim().parse().ok()?; + let day: i8 = parts.next()?.trim().parse().ok()?; + let year: i16 = parts.next()?.trim().parse().ok()?; + + let time = time.trim(); + let (clock, meridiem) = time.rsplit_once(' ')?; + let mut hms = clock.split(':'); + let mut hour: i8 = hms.next()?.parse().ok()?; + let minute: i8 = hms.next()?.parse().ok()?; + let second: i8 = hms.next().unwrap_or("0").parse().ok()?; + match meridiem.to_ascii_uppercase().as_str() { + "PM" if hour < 12 => hour += 12, + "AM" if hour == 12 => hour = 0, + "AM" | "PM" => {} + _ => return None, + } + let dt = jiff::civil::date(year, month, day).at(hour, minute, second, 0); + dt.to_zoned(jiff::tz::TimeZone::UTC) + .ok() + .map(|z| z.timestamp()) +} + +fn parse_opencode_markdown(text: &str, anchor: jiff::Timestamp) -> Vec { + let anchor = opencode_created(text).unwrap_or(anchor); + let mut out: Vec = Vec::new(); + let mut role: Option<&'static str> = None; + let mut body = String::new(); + + let flush = |role: Option<&'static str>, body: &mut String, out: &mut Vec| { + let Some(role) = role else { + body.clear(); + return; + }; + let text = body.trim().to_string(); + body.clear(); + if text.is_empty() { + return; + } + let index = out.len(); + out.push(Message { + agent: "opencode".to_string(), + role: role.to_string(), + text, + created_at: anchor_series(anchor, index), + approximate: true, + }); + }; + + for line in text.lines() { + if let Some(next) = markdown_speaker(line) { + flush(role.take(), &mut body, &mut out); + role = Some(next); + continue; + } + if role.is_some() && line.trim_end() != "---" { + body.push_str(line); + body.push('\n'); + } + } + flush(role.take(), &mut body, &mut out); + out +} + +// --------------------------------------------------------------- scrollback + +/// Parses terminal scrollback pasted into a file. +/// +/// The lossiest path engram offers, and gated behind an explicit `--format` +/// for that reason. There is no structure here — the speaker is a glyph in +/// column zero (`❯` a prompt, `●` a reply, `⎿` a tool result, `✻` a spinner), +/// the text was hard-wrapped to the terminal's width with a two-space +/// continuation indent, and no timestamp survives anywhere. Wrapping cannot be +/// undone faithfully, so continuation lines are re-joined with a space and the +/// original line breaks are simply gone. +fn parse_claude_scrollback(text: &str, anchor: jiff::Timestamp) -> Vec { + let mut out: Vec = Vec::new(); + let mut role: Option<&'static str> = None; + let mut body = String::new(); + + let flush = |role: Option<&'static str>, body: &mut String, out: &mut Vec| { + let Some(role) = role else { + body.clear(); + return; + }; + let text = body.trim().to_string(); + body.clear(); + if text.is_empty() { + return; + } + let index = out.len(); + out.push(Message { + agent: "claude-code".to_string(), + role: role.to_string(), + text, + created_at: anchor_series(anchor, index), + approximate: true, + }); + }; + + for line in text.lines() { + let mut chars = line.chars(); + match chars.next() { + Some('❯') => { + flush(role.take(), &mut body, &mut out); + role = Some("user"); + body.push_str(chars.as_str().trim()); + } + Some('●') => { + flush(role.take(), &mut body, &mut out); + role = Some("assistant"); + body.push_str(chars.as_str().trim()); + } + // Tool results, spinners and status lines are chrome, not speech — + // the same payloads `transcript` excludes by default. + Some('⎿' | '✻' | '⏺') => { + flush(role.take(), &mut body, &mut out); + } + _ if role.is_some() => { + let t = line.trim(); + if !t.is_empty() { + if !body.is_empty() { + body.push(' '); + } + body.push_str(t); + } + } + _ => {} + } + } + flush(role.take(), &mut body, &mut out); + out +} + +/// The stable id of an imported message. +/// +/// Over the message's own content rather than its file, which is what makes +/// every dedupe requirement fall out of `INSERT OR IGNORE` instead of a +/// separate pass: re-importing an archive inserts nothing, thirteen +/// byte-identical copies of one file import once, and an archive holding two +/// concatenated copies of itself (the old `save-chat` append bug, still on +/// disk) yields one set of messages. Scope is included so the same archive +/// filed under two scopes really does land in both. +#[must_use] +pub fn import_id(scope: &str, m: &Message) -> String { + uuid::Uuid::new_v5( + &uuid::Uuid::NAMESPACE_OID, + format!( + "engram-import:{scope}:{}:{}:{}:{}", + m.agent, m.role, m.created_at, m.text + ) + .as_bytes(), + ) + .to_string() +} + +/// Reads and parses one export file. +/// +/// # Errors +/// +/// Propagates I/O failures, refuses a file above `max_bytes`, and returns +/// [`ImportError::Unrecognised`] when no format matches and none was forced. +pub fn read_file( + path: &Path, + forced: Option, + max_bytes: u64, +) -> Result<(Format, Vec), ImportError> { + let meta = std::fs::metadata(path).map_err(ImportError::Io)?; + if meta.len() > max_bytes { + return Err(ImportError::TooLarge { + bytes: meta.len(), + max_bytes, + }); + } + let text = std::fs::read_to_string(path).map_err(ImportError::Io)?; + let format = forced + .or_else(|| sniff(&text)) + .ok_or(ImportError::Unrecognised)?; + let anchor = file_anchor(&meta); + let messages = parse(&text, format, anchor)?; + Ok((format, messages)) +} + +/// The instant a timestamp-less export is anchored to: its own mtime. +/// +/// Not the wall clock. An archive's mtime is at least *about* when the +/// conversation happened, so ordering between files stays roughly right, while +/// `now()` would stack every historical import into the present and sort them +/// by the order they happened to be read. +fn file_anchor(meta: &std::fs::Metadata) -> jiff::Timestamp { + meta.modified() + .ok() + .and_then(|t| jiff::Timestamp::try_from(t).ok()) + .unwrap_or_else(jiff::Timestamp::now) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn anchor() -> jiff::Timestamp { + "2026-01-01T00:00:00Z".parse().expect("anchor") + } + + #[test] + fn unescape_reverses_exactly_three_characters() { + assert_eq!(unescape_texinfo("@@code @{a@} b"), "@code {a} b"); + // A left-to-right scan: the escaped `@@` is consumed before the brace + // rule can see the `{` behind it. + assert_eq!(unescape_texinfo("@@{"), "@{"); + // An `@` that is not an escape survives untouched. + assert_eq!(unescape_texinfo("user@host"), "user@host"); + } + + #[test] + fn heading_parses_both_dialects() { + let v2 = "@chapter Message by claude-code (user) at 2026-08-09T22:18:52.917Z"; + let v1 = "@section Message by codex (assistant) at 2026-07-26T00:04:16.394725123Z"; + assert_eq!( + parse_message_heading(v2), + Some(( + "claude-code".to_string(), + "user".to_string(), + "2026-08-09T22:18:52.917Z".to_string() + )) + ); + assert_eq!( + parse_message_heading(v1).map(|(a, r, _)| (a, r)), + Some(("codex".to_string(), "assistant".to_string())) + ); + // A document title is not a message. + assert_eq!( + parse_message_heading("@chapter Chat history for scope: engram"), + None + ); + } + + /// The third role survives. A corpus of 7,163 archived messages held 136 + /// `note` rows; folding them into `user` or `assistant` would rewrite what + /// the archive said to fit a two-variant type. + #[test] + fn texinfo_keeps_the_note_role() { + let doc = "\\input texinfo\n\ + @chapter Message by test-model (note) at 2026-07-11T08:08:35Z\n\ + Hello from model\n\ + @bye\n"; + let msgs = parse_texinfo(doc).expect("parses"); + assert_eq!(msgs.len(), 1); + assert_eq!(msgs[0].role, "note"); + assert_eq!(msgs[0].text, "Hello from model"); + assert!(!msgs[0].approximate); + } + + /// The old `save-chat` appended instead of rewriting, so one archive on + /// disk holds two copies of the same document. Identity is the content, so + /// the duplicate collapses at insert time. + #[test] + fn a_double_appended_archive_yields_duplicate_ids() { + // The real file interleaves a provenance comment and a title chapter + // between the two copies — that is what used to leak into the body. + let one = "@c Signed by: a-model on 2026-07-11T08:08:39Z\n\ + @chapter Chat history for scope: plan-test\n\ + @section Message by test-model (note) at 2026-07-11T08:08:35Z\n\ + Hello\n"; + let doc = format!("\\input texinfo\n{one}{one}@bye\n"); + let msgs = parse_texinfo(&doc).expect("parses"); + assert_eq!(msgs.len(), 2, "both copies are parsed"); + assert_eq!(msgs[0].text, "Hello", "no structural line leaked in"); + assert_eq!( + import_id("s", &msgs[0]), + import_id("s", &msgs[1]), + "and collapse to one row on INSERT OR IGNORE" + ); + } + + #[test] + fn markdown_ignores_headings_inside_a_reply() { + let doc = "# Project\n\n**Session ID:** ses_x\n\n---\n\n\ + ## User\n\nfix the build\n\n---\n\n\ + ## Assistant (Build · Model · 4.7s)\n\n\ + ## Root Cause Analysis\n\nthe flag moved\n\n\ + ## Next Move\n\nrename it\n\n---\n"; + let msgs = parse_opencode_markdown(doc, anchor()); + assert_eq!(msgs.len(), 2, "one user turn and one assistant turn"); + assert_eq!(msgs[0].role, "user"); + assert!(msgs[1].text.contains("Root Cause Analysis")); + assert!(msgs[1].text.contains("rename it")); + } + + /// Ordering is exact even though the absolute times are not. + #[test] + fn synthesised_timestamps_are_ordered_and_flagged() { + let doc = "**Session ID:** x\n## User\na\n## Assistant\nb\n## User\nc\n"; + let msgs = parse_opencode_markdown(doc, anchor()); + assert_eq!(msgs.len(), 3); + assert!(msgs.iter().all(|m| m.approximate)); + assert!(msgs[0].created_at < msgs[1].created_at); + assert!(msgs[1].created_at < msgs[2].created_at); + } + + #[test] + fn opencode_header_time_beats_the_file_anchor() { + let doc = "**Session ID:** x\n**Created:** 7/7/2026, 9:39:41 PM\n## User\na\n"; + let msgs = parse_opencode_markdown(doc, anchor()); + assert!( + msgs[0].created_at.starts_with("2026-07-07T21:39:41"), + "got {}", + msgs[0].created_at + ); + } + + #[test] + fn scrollback_uses_glyphs_and_drops_chrome() { + let doc = "❯ finish the feature\n because it is missing\n\ + ⎿ Listed directory\n\ + ● I'll start by loading\n the six skills\n\ + ✻ Baked for 5m 21s\n"; + let msgs = parse_claude_scrollback(doc, anchor()); + assert_eq!(msgs.len(), 2); + assert_eq!(msgs[0].role, "user"); + // Hard wrapping is re-joined; the original breaks are unrecoverable. + assert_eq!(msgs[0].text, "finish the feature because it is missing"); + assert_eq!(msgs[1].role, "assistant"); + assert!(!msgs[1].text.contains("Baked for")); + } + + #[test] + fn sniff_recognises_each_family_and_declines_the_rest() { + assert_eq!( + sniff("\\input texinfo\n@bye\n"), + Some(Format::EngramTexinfo) + ); + assert_eq!( + sniff("**Session ID:** x\n## User\nhi\n"), + Some(Format::OpencodeMarkdown) + ); + assert_eq!(sniff("❯ hello\n"), Some(Format::ClaudeScrollback)); + // The palette-editor page that shared a `chat/` directory with real + // archives: an extension check would have taken it, a content check + // does not. + assert_eq!(sniff("\nPalette Lab"), None); + } + + #[test] + fn a_bad_timestamp_is_an_error_not_a_substitution() { + let doc = "\\input texinfo\n@chapter Message by a (user) at not-a-time\nhi\n@bye\n"; + assert!(matches!( + parse_texinfo(doc), + Err(ImportError::BadTimestamp { .. }) + )); + } +} diff --git a/src/main.rs b/src/main.rs index d946f66..47d6194 100644 --- a/src/main.rs +++ b/src/main.rs @@ -11,6 +11,7 @@ mod error; mod facts; mod harness; mod http; +mod import; mod install; mod managed_file; mod mcp; @@ -678,6 +679,24 @@ fn run( ) } + Command::Import { + paths, + scope, + input_format, + recursive, + max_bytes, + dry_run, + } => handle_import( + &store, + mode, + &paths, + scope, + input_format, + recursive, + max_bytes, + dry_run, + ), + Command::Install { harness: requested, db_path, @@ -1553,6 +1572,203 @@ fn fail(err: AppError, mode: OutputMode) -> i32 { err.exit_code } +/// One export that produced messages. +#[derive(serde::Serialize)] +struct ImportedFile { + path: String, + format: &'static str, + scope: String, + messages: usize, + inserted: usize, + skipped_existing: usize, + /// True when the export carried no per-message times and they were + /// synthesised in order. Surfaced per file so a reader can tell an + /// approximate history from a recorded one without opening it. + approximate_times: bool, +} + +/// An export that produced nothing, and why. +#[derive(serde::Serialize)] +struct SkippedFile { + path: String, + reason: String, +} + +/// `engram import` payload. +#[derive(serde::Serialize)] +struct ImportResult { + dry_run: bool, + files: Vec, + /// Never empty on a real corpus: a `chat/` directory accumulates whatever + /// was dropped in it. Reporting each skip with its reason is the + /// difference between "engram found nothing" and "engram found nine files + /// it could not read". + skipped: Vec, + scopes: Vec, + inserted: usize, + skipped_existing: usize, + filtered: transcript::FilterStats, + redactions: transcript::redact::Redactions, +} + +/// Collects the files an import should consider. +/// +/// Walks with `std::fs` and therefore ignores `.gitignore` entirely, which is +/// the only thing that makes this command useful: engram itself adds `chat/` +/// to a project's `.gitignore` the first time `save-chat` runs, so every +/// archive it has ever written is ignored by definition. A walker that +/// honoured ignore rules — as `fd`, `rg` and `git ls-files` all do by default +/// — would find precisely zero of them and report success. +fn collect_exports(paths: &[std::path::PathBuf], recursive: bool) -> Vec { + let mut out = Vec::new(); + let mut stack: Vec = paths.to_vec(); + while let Some(p) = stack.pop() { + if p.is_file() { + out.push(p); + continue; + } + let Ok(entries) = std::fs::read_dir(&p) else { + continue; + }; + for entry in entries.flatten() { + let path = entry.path(); + if path.is_dir() { + if recursive { + stack.push(path); + } + } else { + out.push(path); + } + } + } + out.sort(); + out +} + +/// Imports exported transcripts into the store. +#[allow(clippy::too_many_arguments, clippy::too_many_lines)] +fn handle_import( + store: &std::sync::Mutex, + mode: OutputMode, + paths: &[std::path::PathBuf], + scope: Option, + format: Option, + recursive: bool, + max_bytes: u64, + dry_run: bool, +) -> i32 { + let files = collect_exports(paths, recursive); + let mut result = ImportResult { + dry_run, + files: Vec::new(), + skipped: Vec::new(), + scopes: Vec::new(), + inserted: 0, + skipped_existing: 0, + filtered: transcript::FilterStats::default(), + redactions: transcript::redact::Redactions::default(), + }; + + for path in files { + let display = path.display().to_string(); + let (fmt, messages) = match import::read_file(&path, format, max_bytes) { + Ok(v) => v, + Err(e) => { + result.skipped.push(SkippedFile { + path: display, + reason: e.to_string(), + }); + continue; + } + }; + if messages.is_empty() { + result.skipped.push(SkippedFile { + path: display, + reason: "recognised the format but found no messages".to_string(), + }); + continue; + } + + // Scope per file, resolved from the directory the export sits in, so + // one run over many projects files each into its own scope. + // + // Canonicalized first: `engram import ./chat` resolves its parent to + // `.`, whose basename is empty, and the scope silently became + // `default` — one relative path away from filing an entire corpus + // under the wrong name. + let base = path.parent().unwrap_or(&path); + let base = std::fs::canonicalize(base).unwrap_or_else(|_| base.to_path_buf()); + let resolved = rules::resolve_scope_in(scope.as_deref(), &base); + + let approximate = messages.iter().any(|m| m.approximate); + let mut rows: Vec = Vec::with_capacity(messages.len()); + for m in &messages { + // The same normalization and redaction the live readers apply. An + // archive is not more trustworthy for being old — these files hold + // whatever was pasted into the session. + let Some(text) = transcript::normalize_text( + &m.text, + transcript::DEFAULT_MAX_CHARS_PER_TURN, + &mut result.filtered, + ) else { + continue; + }; + let text = transcript::redact::scrub(&text, &mut result.redactions); + let scrubbed = import::Message { text, ..m.clone() }; + rows.push(store::IngestTurn { + id: import::import_id(&resolved.name, &scrubbed), + agent: scrubbed.agent.clone(), + role: scrubbed.role.clone(), + content: scrubbed.text.clone(), + created_at: scrubbed.created_at.clone(), + }); + } + + let report = if dry_run { + store::IngestReport { + inserted: rows.len(), + skipped_existing: 0, + } + } else { + let mut guard = store.lock().expect("store lock poisoned"); + match guard.ingest_turns(&resolved.name, &rows) { + Ok(r) => r, + Err(e) => return fail(AppError::from(e), mode), + } + }; + + result.inserted += report.inserted; + result.skipped_existing += report.skipped_existing; + if !result.scopes.contains(&resolved.name) { + result.scopes.push(resolved.name.clone()); + } + result.files.push(ImportedFile { + path: display, + format: fmt.as_str(), + scope: resolved.name, + messages: rows.len(), + inserted: report.inserted, + skipped_existing: report.skipped_existing, + approximate_times: approximate, + }); + } + + result.scopes.sort(); + let label = if dry_run { + "engram import --dry-run" + } else { + "engram import" + }; + let response = Response::new(label, result); + let response = if dry_run { + response.with_dry_run() + } else { + response + }; + emit_ok(response, mode); + 0 +} + // Rust guideline compliant 2026-05-18 fn emit_ok(resp: Response, mode: OutputMode) { diff --git a/tests/cli.rs b/tests/cli.rs index 9d40d7f..36689a9 100644 --- a/tests/cli.rs +++ b/tests/cli.rs @@ -3617,3 +3617,192 @@ fn ingest_cwd_sets_the_scope_and_explicit_scope_still_wins() { assert_eq!(data["scope"], "pinned"); assert_eq!(data["scope_origin"], "explicit"); } + +/// An `engram import` invocation against a project directory. +fn import(db: &Path, project: &Path, args: &[&str]) -> Command { + let mut cmd = engram(db); + cmd.current_dir(project); + cmd.arg("import"); + cmd.args(args); + cmd +} + +/// Writes a small engram archive into `/chat/`. +fn plant_archive(project: &Path, name: &str, body: &str) -> std::path::PathBuf { + let dir = project.join("chat"); + std::fs::create_dir_all(&dir).expect("create chat dir"); + let path = dir.join(name); + std::fs::write(&path, body).expect("write archive"); + path +} + +const ARCHIVE: &str = "\\input texinfo @c -*-texinfo-*-\n\ + @documentencoding UTF-8\n\ + @settitle Chat history for scope: whatever\n\n\ + @c Generated by `engram save-chat`. Edits are overwritten.\n\ + @node Top\n\ + @top Chat history for scope: whatever\n\n\ + @chapter Message by claude-code (user) at 2026-07-26T00:04:16.394Z\n\ + please fix the flag\n\n\ + @chapter Message by claude-code (assistant) at 2026-07-26T00:05:38.892Z\n\ + Renamed it and re-ran the gate.\n\n\ + @chapter Message by test-model (note) at 2026-07-26T00:06:00.000Z\n\ + archived for later\n\ + @bye\n"; + +/// Round-trips what `save-chat` writes, which engram could not read back. +/// +/// The scope comes from the directory the archive sits in, not from the +/// `@settitle` line — recorded scopes in the real corpus are inconsistent +/// (bare names, paths, absolute paths, branch names), so the containing +/// project is the reliable signal. +#[test] +fn import_reads_an_engram_archive_and_is_idempotent() { + let tmp = TempDir::new().expect("tempdir"); + let db = tmp.path().join("test.db"); + let project = pinned_project(&tmp); + plant_archive(&project, "2026-07-26T00-00-00Z.texi", ARCHIVE); + + let assert = import(&db, &project, &["chat"]).assert().success(); + let data = parse_single_line_json(&assert.get_output().stdout)["data"].clone(); + assert_eq!(data["inserted"], 3); + assert_eq!( + data["scopes"][0], "project", + "scope is the containing project" + ); + assert_eq!(data["files"][0]["format"], "engram-texinfo"); + assert_eq!(data["files"][0]["approximate_times"], false); + + // The third role survives rather than being coerced. + let assert = engram(&db) + .args(["recall", "--scope", "project", "--limit", "10"]) + .assert() + .success(); + let roles: Vec = parse_single_line_json(&assert.get_output().stdout)["data"] + .as_array() + .expect("memories") + .iter() + .map(|m| m["role"].as_str().unwrap_or_default().to_string()) + .collect(); + assert!(roles.contains(&"note".to_string()), "roles: {roles:?}"); + assert!(roles.contains(&"user".to_string())); + assert!(roles.contains(&"assistant".to_string())); + + // Identity is the content, so a second run writes nothing. + let assert = import(&db, &project, &["chat"]).assert().success(); + let data = parse_single_line_json(&assert.get_output().stdout)["data"].clone(); + assert_eq!(data["inserted"], 0); + assert_eq!(data["skipped_existing"], 3); +} + +/// A file that is not a transcript is skipped with a reason, not an error. +/// +/// A `chat/` directory accumulates whatever was dropped in it — the real one +/// held a standalone HTML palette editor alongside real archives. Sniffing +/// content rather than extension is what keeps that out, and reporting the +/// skip is what distinguishes "found nothing" from "could not read nine +/// files". +#[test] +fn import_skips_what_it_cannot_recognise_and_says_so() { + let tmp = TempDir::new().expect("tempdir"); + let db = tmp.path().join("test.db"); + let project = pinned_project(&tmp); + plant_archive(&project, "real.texi", ARCHIVE); + plant_archive( + &project, + "palette.html", + "\nPalette Lab\n\n", + ); + + let assert = import(&db, &project, &["chat"]).assert().success(); + let data = parse_single_line_json(&assert.get_output().stdout)["data"].clone(); + + assert_eq!(data["files"].as_array().expect("files").len(), 1); + let skipped = data["skipped"].as_array().expect("skipped"); + assert_eq!(skipped.len(), 1); + assert!( + skipped[0]["path"] + .as_str() + .expect("path") + .ends_with(".html"), + "{skipped:?}" + ); + assert!( + skipped[0]["reason"] + .as_str() + .expect("reason") + .contains("no known export format"), + "{skipped:?}" + ); +} + +/// One run over several projects files each into its own scope. +#[test] +fn import_files_each_project_into_its_own_scope() { + let tmp = TempDir::new().expect("tempdir"); + let db = tmp.path().join("test.db"); + + for name in ["alpha", "beta"] { + let project = tmp.path().join(name); + std::fs::create_dir_all(&project).expect("create project"); + std::fs::write(project.join(".git"), "gitdir: elsewhere\n").expect("pin git root"); + plant_archive(&project, "a.texi", ARCHIVE); + } + + let here = pinned_project(&tmp); + let alpha = tmp.path().join("alpha").to_string_lossy().into_owned(); + let beta = tmp.path().join("beta").to_string_lossy().into_owned(); + let assert = import(&db, &here, &[&alpha, &beta, "--recursive"]) + .assert() + .success(); + let data = parse_single_line_json(&assert.get_output().stdout)["data"].clone(); + + let mut scopes: Vec = data["scopes"] + .as_array() + .expect("scopes") + .iter() + .map(|s| s.as_str().unwrap_or_default().to_string()) + .collect(); + scopes.sort(); + assert_eq!(scopes, vec!["alpha".to_string(), "beta".to_string()]); +} + +/// Timestamp-less exports are ordered, flagged, and still importable. +#[test] +fn import_synthesises_ordered_times_for_a_markdown_export() { + let tmp = TempDir::new().expect("tempdir"); + let db = tmp.path().join("test.db"); + let project = pinned_project(&tmp); + plant_archive( + &project, + "session.md", + "# Project\n\n**Session ID:** ses_x\n**Created:** 7/7/2026, 9:39:41 PM\n\n---\n\n\ + ## User\n\nfix the build\n\n---\n\n\ + ## Assistant (Build · Model · 4.7s)\n\n\ + ## Root Cause Analysis\n\nthe flag moved\n\n---\n", + ); + + let assert = import(&db, &project, &["chat"]).assert().success(); + let data = parse_single_line_json(&assert.get_output().stdout)["data"].clone(); + assert_eq!(data["files"][0]["format"], "opencode-markdown"); + assert_eq!(data["files"][0]["approximate_times"], true); + assert_eq!( + data["inserted"], 2, + "a heading inside a reply is not a turn" + ); + + // Order is exact even though the absolute times are not. + let assert = engram(&db) + .args(["recall", "--scope", "project"]) + .assert() + .success(); + let mems = parse_single_line_json(&assert.get_output().stdout)["data"].clone(); + let times: Vec<&str> = mems + .as_array() + .expect("memories") + .iter() + .map(|m| m["created_at"].as_str().unwrap_or_default()) + .collect(); + assert!(times[0] < times[1], "{times:?}"); + assert!(times[0].starts_with("2026-07-07T21:39:41"), "{times:?}"); +}