diff --git a/CHANGELOG.md b/CHANGELOG.md index 7f4149f..9290cc1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -39,20 +39,16 @@ releases may contain breaking changes. represent — a lone surrogate, which only an API assignment can introduce — is refused with `LiftWriteError` naming the node, and reported by validation as `lone-surrogate`. +- Generated timestamps on save: `Lexicon.save()` and `Lexicon.save_zip()` stamp + `dateModified` on every entry whose content changed since it was loaded, and + fill a blank `dateCreated` with the same moment. `stamp=False` turns it off; + `when=` supplies the moment in place of the wall clock. - Change detection against the loaded document, reading the same parse-time - digests. `Lexicon.changed_entries()` reports entries whose content differs - (an entry's digest covers its whole subtree, so an edit at any depth reports - the entry containing it, while an identical rewrite or a `sort()` reports - nothing); `added_entries()` and `removed_entries()` report the rest — an - entry already in the document, appended a second time, counts as an addition - — and need no serialization. `Lexicon.changes()` and `RangesFile.changes()` - return `Changes` / `RangesChanges`, covering entry content, additions, - removals, reordering, the header, the root element, and every tracked - companion — each falsy only when `save()` would reproduce the source bytes, - so `if not lex.changes():` is a correct guard for skipping an in-place write - (content, not destination: a `save(path)` into another directory writes - there regardless). Comparison is always against the loaded document, never - against the most recent `save()`. + digests. `Lexicon.changed_entries()`, `added_entries()`, and + `removed_entries()` report entries; `Lexicon.changes()` and + `RangesFile.changes()` return a `Changes` / `RangesChanges` summary covering + entry content, additions, removals, reordering, the header, the root + element, and tracked companions. - LIFT-folder handling: `RangesFile` (standalone `.lift-ranges` documents, same fidelity guarantees), automatic companion discovery/tracking on load (`Lexicon.ranges_files`), `save()` writes companions together, diff --git a/README.md b/README.md index f9796ef..c004e8d 100644 --- a/README.md +++ b/README.md @@ -8,12 +8,12 @@ canonical sorting — with streaming APIs for large lexicons. ```python import sil_lift -lex = sil_lift.load("thesaurus.lift") # tracks .lift-ranges companions too +lex = sil_lift.load("thesaurus.lift") # tracks .lift-ranges companions too for entry in lex.entries: ... entry = lex.find(id="hoofd_a1b2") entry.senses[0].definition["en"] = "head (anatomy)" -lex.save() # untouched entries byte-identical +lex.save() # edited entries stamped; rest verbatim ``` **Status: pre-release, under active development.** The API is not yet stable. @@ -35,7 +35,9 @@ not understand. - Entries you did not modify are written back **byte-identical**, even when other entries changed. - Entries you did modify are re-serialized in a documented canonical form, with - all out-of-schema content (unknown elements, attributes, comments) preserved. + all out-of-schema content (unknown elements, attributes, comments) preserved, + and are stamped with a fresh `dateModified` so the edit does not go out under + the date it was loaded with. - Whitespace inside `` is never altered — not even for indentation. The precise rules and their few edge cases are documented in diff --git a/docs/en/fidelity.md b/docs/en/fidelity.md index d217429..40ef90e 100644 --- a/docs/en/fidelity.md +++ b/docs/en/fidelity.md @@ -8,7 +8,7 @@ Any well-formed LIFT 0.13 document loads — schema-invalid content included. Wh ## Saving an unchanged document -`load()` → `save()` with no edits writes **byte-identical output** — no reformatting, no re-escaping, no reordering, byte-order marks and XML declarations included. There is currently no normalization list: identity is exact. +`load()` → `save()` with no edits writes **byte-identical output** — no reformatting, no re-escaping, no reordering, byte-order marks and XML declarations included. There is currently no normalization list: identity is exact. Timestamps are generated from content, so an unchanged document has none generated either. Exceptions (the writer falls back to full canonical serialization, which is semantically complete but not byte-preserving): @@ -20,12 +20,33 @@ Exceptions (the writer falls back to full canonical serialization, which is sema ## Saving an edited document - **Untouched entries are emitted verbatim from their original bytes.** An entry counts as touched if any part of its model object changed since parse (detected by canonical-serialization snapshot, not a dirty flag). -- **Touched entries are re-serialized canonically and completely**: UTF-8, 2-space indentation _outside_ mixed content (whitespace inside `` and `` is never altered), a documented child grouping per element (e.g. entry: lexical-unit, citation, pronunciations, variants, senses, notes, relations, etymologies, annotations, traits, fields), fixed attribute order, dates in ISO-8601 (`Z` for UTC). All residue is re-emitted; its position is restored to the original child index, clamped to the new child list (an approximation — exact byte positions are only guaranteed for untouched entries). +- **Touched entries are re-serialized canonically and completely.** Canonical form is: + - UTF-8. + - 2-space indentation _outside_ mixed content; whitespace inside `` and `` is never altered. + - A documented child grouping per element; for ``: lexical-unit, citation, pronunciations, variants, senses, notes, relations, etymologies, annotations, traits, fields. + - Fixed attribute order. + - Dates in ISO-8601 (`Z` for UTC). +- **All residue is re-emitted.** Its position is restored to the original child index, clamped to the new child list — an approximation, since exact byte positions are only guaranteed for untouched entries. - Adding, removing, or reordering entries re-serializes the document structure but still emits every unchanged entry's bytes verbatim. +- **A touched entry is stamped** with a fresh `dateModified`, and a `dateCreated` if it had none — see [Generated timestamps](#generated-timestamps). !!! note ""Canonical" here is not related to any other Canonical XML" Canonical form on this page means `sil-lift`'s own documented shape, described in a bullet above. It is unrelated to W3C's Canonical XML (C14N) process. It is unrelated to `SIL.Core`'s `CanonicalXmlSettings` class. +## Generated timestamps + +A generated stamp is the one thing in the output that is not a function of the input. + +- **What is stamped.** Every entry whose content changed since it was read, with a fresh `dateModified` and, if it had none, a `dateCreated` of the same moment. An edit shipped under its loaded date looks unmodified to everything that reconciles on that attribute, FieldWorks and The Combine's LIFT import included. +- **Entries only.** No node below an ``, and nothing in the header. +- **What is left alone.** An entry whose date the caller set deliberately, and an entry created since the load that already carries one. +- **Clearing a date is honored.** `entry.date_modified = None` on an entry read with one is a deliberate value like any other, so the entry goes out with no `dateModified` whether or not its content moved. An entry read without one is a different case: `None` there is the value already in place, indistinguishable from never touching the field, so an edit still stamps it. +- **An unparseable date is replaced.** A date the model could not parse is [residue](#reading) rather than a date, so a stamp overwrites it and the original string is dropped — an edited entry is better off carrying a real date than `dateModified="whenever"`. +- **The moment.** UTC at seconds precision (`YYYY-MM-DDTHH:MM:SSZ`, the shape every surveyed FieldWorks export uses), read from the wall clock. One second holds one date, so an edit saved within a second of the previous one carries the same stamp. +- **`save(when=...)`** supplies the moment instead of the wall clock, which is what keeps stamped output reproducible for a diff-based CI gate. It must be timezone-aware, and is normalized to UTC whole seconds. +- **`save(stamp=False)`** writes the model exactly as it stands, residue included. +- **Stamping commits with the `.lift` write.** Anything that keeps that write from landing puts the dates back. Once it lands they stand, even if a companion write fails after it. + ## Content XML cannot represent Non-BMP characters — emoji, CJK Extension B, Adlam, anything above U+FFFF — are ordinary content and round-trip byte-identically. A "surrogate pair" is a UTF-16 encoding detail: Python strings are sequences of codepoints, so nothing in the reader, the byte scanner, or the writer ever sees one. diff --git a/docs/en/guides/build-export.md b/docs/en/guides/build-export.md index ce13b26..a299eb9 100644 --- a/docs/en/guides/build-export.md +++ b/docs/en/guides/build-export.md @@ -52,7 +52,7 @@ ranges.add_range("grammatical-info").add_element("Noun").label["en"] = "noun" ranges.add_range("semantic-domain-ddp4").add_element("1.6.1.2").label["en"] = "Bird" lex.add_ranges_file(ranges, href="birds.lift-ranges") -# Validate what save() would write, before touching the disk. +# Validate the document as it stands, before touching the disk. problems = list(lex.iter_problems()) print(f"validation: {len(problems)} problem(s)") @@ -79,7 +79,7 @@ print((out / "birds.lift-ranges").read_text(encoding="utf-8"), end="") - +
nkhuku @@ -161,7 +161,8 @@ print((out / "birds.lift-ranges").read_text(encoding="utf-8"), end="") - A `URLRef` is an href plus an optional caption/label multitext — used for both `` (audio) and `` (photos). The pronunciation here follows The Combine's convention of an `en` form reading `Speaker: `. - App-specific data with no native LIFT home rides as a `` (or ``): FieldWorks reads these as custom fields and The Combine preserves them. - Give every entry a real, stable `guid` (e.g. from `uuid.uuid4()`, reused across exports) — a later re-import updates the entry in place rather than duplicating it. `sil-lift validate --require-ids` enforces this. -- `lex.iter_problems()` validates the in-memory document (what `save()` would write) before anything hits disk; here it is clean. Because the lexicon has no folder yet, the media-presence and companion-href checks are skipped — run [`sil-lift validate`](cli.md) on the saved output (or with `--no-check-media`) once the audio and photo files are in place. +- The `dateCreated`/`dateModified` in the output above are not in the script: `save()` stamped them with the moment it ran, since an entry written for the first time carries no date of its own. `when=` supplies that moment instead of reading the clock, which is what makes a generated export byte-reproducible for a CI job to diff; `stamp=False` writes no dates at all. See [Generated timestamps](../fidelity.md#generated-timestamps) for the rest of the contract. +- `lex.iter_problems()` validates the in-memory document before anything hits disk; here it is clean. Because the lexicon has no folder yet, the media-presence and companion-href checks are skipped — run [`sil-lift validate`](cli.md) on the saved output (or with `--no-check-media`) once the audio and photo files are in place. ## Packaging diff --git a/docs/en/guides/bulk-edit-glosses.md b/docs/en/guides/bulk-edit-glosses.md index 2a82496..d784a4b 100644 --- a/docs/en/guides/bulk-edit-glosses.md +++ b/docs/en/guides/bulk-edit-glosses.md @@ -52,6 +52,7 @@ A few things worth noting: - It compares content, not destination, so guard only an in-place save with it: `lex.save(some_other_dir / "dictionary.lift")` writes the document and its companions to a location that has nothing in it yet, whether or not anything changed. - It is a guard, not a speed-up — answering it digests every entry, which is the same work `save()` does to decide which source bytes it can reuse, so what you skip is the write itself (an unchanged file-modification time, no spurious diff), not the effort of deciding. - Validating in memory (`lex.iter_problems()`) serializes the edited state first, so it correctly reflects the edit before anything is written to disk. Aborting on any `"error"`-level `Problem` — warnings are left for the caller to decide about — means a bad edit never reaches `save()`. +- `lex.save()` stamps a fresh `dateModified` on exactly the entries this script changed, filling `dateCreated` where it was blank. Without it a lexicon this script has rewritten would look untouched to FieldWorks or to The Combine's LIFT import, both of which decide what to update from that attribute — and it stamps `` only, however deep the edit went: the entry containing the edited subsense gloss, not the sense. Glosses aren't the only thing worth touching this way. The same `Multitext` mapping surface applies to definitions and every other multilingual field on an entry or sense: diff --git a/docs/en/guides/large-files.md b/docs/en/guides/large-files.md index f2bf498..c2dc252 100644 --- a/docs/en/guides/large-files.md +++ b/docs/en/guides/large-files.md @@ -24,4 +24,5 @@ Notes: - The writer's output is exactly what the full-document canonical serializer would produce for the same content — the two modes never drift apart. - Streaming mode reuses no source bytes: output is always canonical. Root-level LIFT residue — comments between entries and out-of-schema attributes on `` — is not carried; entries and the header are complete, residue included. +- Nor does it generate timestamps, as [`Lexicon.save()`](../fidelity.md#saving-an-edited-document) does. An entry is written with the dates it carries: a streaming pass never sees the document as it was, so it cannot tell which entries you changed. Set `entry.date_modified` yourself on the ones this pass rewrites. - If the body of an `open_writer` block raises, the file is left visibly unterminated (no closing ``) — a half-written lexicon must not look complete. diff --git a/docs/en/guides/lift-export-interop.md b/docs/en/guides/lift-export-interop.md index 046a8e8..f5e3c84 100644 --- a/docs/en/guides/lift-export-interop.md +++ b/docs/en/guides/lift-export-interop.md @@ -14,6 +14,7 @@ LIFT is usually moved around as a single `.zip` — FieldWorks and The Combine b - Extraction is capped at 10 GiB and 100,000 members; a package over either limit is refused with a `LiftParseError`, as is one whose member paths escape the extraction directory. - **Write:** `Lexicon.save_zip("out.zip", wrap_folder="MyDict")` packages the `.lift`, its `.lift-ranges`, and every other file in the source folder (media, `WritingSystems/`, `consent/`, ...) into a zip. - `wrap_folder` defaults to a top-level folder named after the zip (the FieldWorks/Combine import convention); pass `False` for a flat archive. + - Entries whose content changed since the load are stamped with a fresh `dateModified` on the way out, exactly as on `save()`: a package is what an importing tool reconciles from, so a stale date there is what makes an updated lexicon look untouched. `stamp=False` and `when=` work the same way here. The `.lift` and `.lift-ranges` keep their byte-fidelity inside the package; the zip container itself is not byte-reproducible. diff --git a/docs/en/guides/read-edit-write.md b/docs/en/guides/read-edit-write.md index 09984e2..9157ee3 100644 --- a/docs/en/guides/read-edit-write.md +++ b/docs/en/guides/read-edit-write.md @@ -49,6 +49,8 @@ lex.save("elsewhere.lift") Entries you didn't modify are written back **byte-identical**; a document you didn't modify at all is byte-identical from the first byte to the last. See [Fidelity guarantees](../fidelity.md) for the precise contract. +The entries you did modify go out with a fresh `dateModified` (and a `dateCreated` if they had none), so an edit doesn't ship under the date it was loaded with — the tools that merge LIFT decide what changed from that attribute. `lex.save(stamp=False)` writes the dates the model holds and nothing more; `lex.save(when=...)` pins the moment instead of reading the clock. See [Generated timestamps](../fidelity.md#generated-timestamps) for the rest of the rules. + ## Building from scratch ```python diff --git a/docs/en/guides/validate.md b/docs/en/guides/validate.md index 87f0733..363f5c9 100644 --- a/docs/en/guides/validate.md +++ b/docs/en/guides/validate.md @@ -46,7 +46,7 @@ Every finding carries one of these, whichever layer produced it — `schema` and | `undefined-range-value` | warning | a grammatical-info or range-keyed trait value the range does not list | | `uri-not-rfc` | warning | an href that is not a valid URI — FLEx's `file://C:/...` | -All three layers work from what `save()` would write, so a document that cannot be serialized at all is reported as a single `lone-surrogate` error instead — see [Fidelity guarantees](../fidelity.md#content-xml-cannot-represent). +All three layers work from the document serialized as it stands, so one that cannot be serialized at all is reported as a single `lone-surrogate` error instead — see [Fidelity guarantees](../fidelity.md#content-xml-cannot-represent). Validation is read-only: it reports the document as it stands, before the `dateModified` stamping a save does. Nothing generated is ever a finding, so validate-then-save is sound. A companion name matching several files loads none of them: the ranges they define go absent until all but one is renamed or removed. diff --git a/docs/en/index.md b/docs/en/index.md index 889b6e4..4f0ca96 100644 --- a/docs/en/index.md +++ b/docs/en/index.md @@ -28,5 +28,5 @@ for entry in lex.entries: entry = lex.find(guid="0f5a9c3e-...") # or lex.find(id="hoofd_a1b2") entry.senses[0].definition["en"] = "head (anatomy)" -lex.save() # untouched entries byte-identical; edited entry re-serialized +lex.save() # edited entry re-serialized and re-stamped; the rest byte-identical ``` diff --git a/src/sil_lift/_canonical.py b/src/sil_lift/_canonical.py index 10c08e3..54e17e3 100644 --- a/src/sil_lift/_canonical.py +++ b/src/sil_lift/_canonical.py @@ -72,6 +72,10 @@ def canonicalize(src: str | os.PathLike[str], dst: str | os.PathLike[str]) -> No diff cleanly. Text content is never whitespace-normalized. The whole document is held in memory (sorting requires it; the C# oracle buffers too). + No timestamp is generated either: sorting and reformatting change no entry's + content, so nothing here is a modification to stamp. The output is a pure + function of the input. + Only the ``.lift`` file is written: companion ``.lift-ranges`` files are neither read nor rewritten (the source is loaded with ``resolve_ranges=False``). Sort a ranges file separately via diff --git a/src/sil_lift/_model.py b/src/sil_lift/_model.py index 94b29b7..43c9b61 100644 --- a/src/sil_lift/_model.py +++ b/src/sil_lift/_model.py @@ -28,7 +28,7 @@ from typing import Literal from ._validate import Problem - from ._writer import _RangesSourceInfo, _SourceInfo + from ._writer import _EntryRecord, _RangesSourceInfo, _SourceInfo, _StampUndo __all__ = [ "Changes", @@ -63,6 +63,20 @@ class _ExtensibleNoFields: traits: list[Trait] = field(default_factory=list) extra: Extras = field(default_factory=Extras) + def _stamp(self, when: datetime) -> None: + """Record ``when`` as this node's modification moment. + + A blank ``dateCreated`` is filled with the same moment: blank beside a + fresh ``dateModified``, it reads as a node modified before it existed. + + The one place either date is generated, so that the nine date-bearing + types share one policy — though only :class:`Entry` is stamped today + (see :meth:`Lexicon.save`). + """ + self.date_modified = when + if self.date_created is None: + self.date_created = when + @dataclass(slots=True, kw_only=True) class _Extensible(_ExtensibleNoFields): @@ -588,6 +602,7 @@ class Lexicon: __slots__ = ( "_source", + "_stamps", "_tempdir", "entries", "extra", @@ -613,6 +628,7 @@ def __init__( self.extra = extra if extra is not None else Extras() self.ranges_files: dict[Path, RangesFile] = {} self._source: _SourceInfo | None = None # set by the reader + self._stamps: dict[int, _EntryRecord] = {} # stamping baselines (see stamp_entries) self._tempdir: tempfile.TemporaryDirectory[str] | None = None # zip extraction, if any @classmethod @@ -672,7 +688,26 @@ def _resolve_ranges(self) -> None: continue self.ranges_files[resolved] = RangesFile.load(found) - def save(self, path: str | os.PathLike[str] | None = None) -> None: + def _apply_stamps(self, stamp: bool, when: datetime | None) -> _StampUndo: + """The stamping step shared by :meth:`save` and :meth:`save_zip`. + + Returns what undoes the pass, so a write that does not go through can + leave the model as it found it. A save that stamps nothing still notes + the dates it writes, so the next stamping save measures against them. + """ + from ._writer import note_caller_dates, resolve_when, stamp_entries + + if not stamp: + return note_caller_dates(self) + return stamp_entries(self, resolve_when(when)) + + def save( + self, + path: str | os.PathLike[str] | None = None, + *, + stamp: bool = True, + when: datetime | None = None, + ) -> None: """Write the ``.lift`` file and every tracked ``.lift-ranges`` companion. Untouched entries are emitted byte-identical to the source; modified @@ -683,37 +718,86 @@ def save(self, path: str | os.PathLike[str] | None = None) -> None: name in the *same* directory leaves companions at their original paths (they are shared with the original document, not copied). + This mutates the model: every entry whose content changed since the + load goes out with a fresh ``dateModified``, and with a ``dateCreated`` + if it had none — in a ``save(path)`` used to export a copy too. Left as + they stand: a date the caller set deliberately, and any entry whose + content did not change, reordering included (see :meth:`sort`). + + ``stamp=False`` writes the model exactly as it stands. ``when`` supplies + the moment in place of the clock, normalized to UTC at seconds + precision, which is what makes stamped output byte-reproducible. See + ``docs/en/fidelity.md`` for the rest of the contract. + Raises :class:`ValueError` if no target path is available (none was - passed and the lexicon was not loaded from a file), and - :class:`~sil_lift.LiftWriteError` if the model holds content XML cannot - represent (a lone surrogate) — nothing is written in that case. + passed and the lexicon was not loaded from a file) or if ``when`` is + naive, and :class:`~sil_lift.LiftWriteError` if the lexicon or one of + its companions holds content XML cannot represent (a lone surrogate) — + nothing is written in that case, and nothing is stamped. """ - from ._writer import render_document + from ._writer import render_document, render_ranges_document target = Path(path) if path is not None else self.path if target is None: raise ValueError("no target path: pass save(path) or load the lexicon from a file") original_dir = self.path.parent if self.path is not None else None - target.write_bytes(render_document(self)) + undo = self._apply_stamps(stamp, when) + written = False + try: + # Rendered ahead of any write, so content refused in a companion + # leaves the .lift untouched too. + lift_bytes = render_document(self) + companions = [ + (ranges_file, dest, render_ranges_document(ranges_file)) + for ranges_file, dest in self._companion_targets(target, original_dir) + ] + target.write_bytes(lift_bytes) + written = True + finally: + # A companion failing further down leaves the .lift on disk carrying + # these stamps, so from here on they stand. + if not written: + undo.restore() self.path = target + for ranges_file, dest, data in companions: + dest.write_bytes(data) + ranges_file.path = dest + # Keys must keep tracking the companions' current locations. + self.ranges_files = { + ranges_file.path.resolve(): ranges_file + for ranges_file in self.ranges_files.values() + if ranges_file.path is not None + } + + def _companion_targets( + self, target: Path, original_dir: Path | None + ) -> list[tuple[RangesFile, Path]]: + """Where each tracked companion goes for a ``.lift`` saved to ``target``. + + Settled before anything is rendered, so the write step is nothing but + bytes to paths (see :meth:`save`). + """ relocating = not _same_dir(target.parent, original_dir) + targets: list[tuple[RangesFile, Path]] = [] for key, ranges_file in self.ranges_files.items(): if ranges_file.path is None: # A from-scratch companion (see add_ranges_file): the dict key # is its intended href — write it beside the saved .lift. - ranges_file.save(target.parent / Path(key).name) + targets.append((ranges_file, target.parent / Path(key).name)) elif relocating: - ranges_file.save(target.parent / ranges_file.path.name) + targets.append((ranges_file, target.parent / ranges_file.path.name)) else: - ranges_file.save() - # Keys must keep tracking the companions' current locations. - self.ranges_files = { - ranges_file.path.resolve(): ranges_file - for ranges_file in self.ranges_files.values() - if ranges_file.path is not None - } + targets.append((ranges_file, ranges_file.path)) + return targets - def save_zip(self, path: str | os.PathLike[str], *, wrap_folder: str | bool = True) -> None: + def save_zip( + self, + path: str | os.PathLike[str], + *, + wrap_folder: str | bool = True, + stamp: bool = True, + when: datetime | None = None, + ) -> None: """Write the lexicon and its folder companions as a zip package. The ``.lift`` and ``.lift-ranges`` are (re-)serialized with the usual @@ -724,10 +808,20 @@ def save_zip(self, path: str | os.PathLike[str], *, wrap_folder: str | bool = Tr convention FieldWorks and The Combine expect on import — ``False`` writes the files at the archive root, and a string uses that folder name. The archive container itself is not byte-reproducible. + + ``stamp`` and ``when`` work exactly as on :meth:`save`, and matter more + here: a package is the hand-off to the tools that read ``dateModified``. """ from ._zip import save_zip - save_zip(self, Path(path), wrap_folder=wrap_folder) + undo = self._apply_stamps(stamp, when) + written = False + try: + save_zip(self, Path(path), wrap_folder=wrap_folder) + written = True + finally: + if not written: + undo.restore() def sort(self) -> None: """Sort into canonical order, in place: entries by (guid, id), header @@ -898,11 +992,17 @@ def changes(self) -> Changes: def iter_problems(self, *, require_ids: bool = False) -> Iterator[Problem]: """Validate the in-memory state (schema layers + semantic checks). - The schema layers need serialized bytes: what :meth:`save` would - write is validated, so in-memory edits are always visible. For an - untouched loaded document those are the source bytes (line numbers - match the file on disk); otherwise serialization is a documented - cost on large lexicons. + The schema layers need serialized bytes, so the document is serialized + as it stands and those bytes are validated: in-memory edits are always + visible. For an untouched loaded document they are the source bytes + (line numbers match the file on disk); otherwise serialization is a + documented cost on large lexicons. + + Read-only: it reports the document as it stands, before the + ``dateModified`` stamping :meth:`save` does. That is the one way these + bytes differ from the bytes written. A stamp is a well-formed date in a + valid place, so nothing generated is ever a finding and validate-then-save + is sound. With ``require_ids``, entries missing a ``guid`` and senses missing an ``id`` are reported as ``missing-id`` errors — stricter than LIFT (both diff --git a/src/sil_lift/_reader.py b/src/sil_lift/_reader.py index 0eff1e9..146279c 100644 --- a/src/sil_lift/_reader.py +++ b/src/sil_lift/_reader.py @@ -59,6 +59,8 @@ def parse_document(path: Path) -> Lexicon: raise LiftParseError(f"{path}: not well-formed XML: {exc}") from exc lexicon = parse_root(root, path=path) _attach_source(lexicon, data, root) + if lexicon._source is None: + _attach_stamp_baseline(lexicon) return lexicon @@ -110,6 +112,26 @@ def _attach_ranges_source(ranges_file: RangesFile, data: bytes, root: etree._Ele ) +def _attach_stamp_baseline(lexicon: Lexicon) -> None: + """Record what a stamping save measures against, for a document with no snapshot. + + Byte reuse needs the source bytes; stamping needs only the digests, which + are available whether or not the scan was declined. Without this a document + that was read rather than built would reach a save with no baseline at all. + Every undated entry would look new, so a save that changed nothing would + stamp entries nobody touched. + + Digesting refuses only a lone surrogate, which cannot arrive from a file, so + this cannot raise for a document that has just parsed. + """ + from ._writer import _EntryRecord, entry_digest + + lexicon._stamps = { + id(entry): _EntryRecord(entry, entry_digest(entry), entry.date_modified) + for entry in lexicon.entries + } + + def _attach_source(lexicon: Lexicon, data: bytes, root: etree._Element) -> None: """Capture the original bytes needed for byte reuse; on any doubt, capture nothing. @@ -141,7 +163,10 @@ def _attach_source(lexicon: Lexicon, data: bytes, root: etree._Element) -> None: root_open_end=result.root_open_end, root_self_closing=result.root_self_closing, children=result.children, - entry_records=[_EntryRecord(entry, entry_digest(entry)) for entry in lexicon.entries], + entry_records=[ + _EntryRecord(entry, entry_digest(entry), entry.date_modified) + for entry in lexicon.entries + ], header_digest=header_digest(lexicon.header) if header_regions else None, producer=lexicon.producer, root_extra_attrs=dict(lexicon.extra._attrs), diff --git a/src/sil_lift/_writer.py b/src/sil_lift/_writer.py index dff61c7..3793ea3 100644 --- a/src/sil_lift/_writer.py +++ b/src/sil_lift/_writer.py @@ -14,7 +14,9 @@ are re-serialized canonically. A fully-unchanged document therefore reassembles byte-identically. -Snapshots are sha256 digests of canonical bytes, taken at parse time. +Snapshots are sha256 digests of canonical bytes, taken at parse time. The same +digests decide, in :func:`stamp_entries`, which entries an edit has left with a +stale ``dateModified``. Both paths refuse content XML cannot represent: see :func:`_guarded`. """ @@ -24,6 +26,7 @@ import hashlib from collections import Counter from dataclasses import dataclass +from datetime import UTC, date, datetime from typing import TYPE_CHECKING from lxml import etree @@ -53,7 +56,6 @@ if TYPE_CHECKING: from collections.abc import Callable, Iterable - from datetime import date, datetime from ._extras import _ExtraNode from ._scan import ChildRegion @@ -61,12 +63,16 @@ __all__ = [ "canonical_document", "canonical_ranges_document", + "default_now", "entry_digest", "header_digest", "node_diff", + "note_caller_dates", "range_digest", "render_document", "render_ranges_document", + "resolve_when", + "stamp_entries", ] _FRAGMENT_PARSER = etree.XMLParser(resolve_entities=False, no_network=True) @@ -105,8 +111,17 @@ def _entry_label(entry: Entry) -> str: @dataclass(slots=True) class _EntryRecord: + """An entry's canonical digest and ``dateModified`` as of one baseline moment. + + The reader builds one per entry at parse time, and those drive byte reuse + and change detection. :func:`stamp_entries` builds a second set as it + saves, so the baseline the stamping policy measures against moves forward + with each save while the parse-time one stays where it is. + """ + entry: Entry # strong ref: keeps id() stable for the identity check digest: bytes + date_modified: datetime | date | None @dataclass(slots=True) @@ -153,6 +168,192 @@ def range_digest(range_: Range) -> bytes: return hashlib.sha256(canonical_range_bytes(range_)).digest() +# --- generated timestamps ------------------------------------------------------- + + +def default_now() -> datetime: + """The clock for generated timestamps: UTC at seconds precision. + + Seconds precision is the shape real exports use: every date literal in the + surveyed FieldWorks 8.3-9.0 exports is the 20-character + ``YYYY-MM-DDTHH:MM:SSZ`` form, with no bare dates, offsets, or fractional + seconds. One second holds one date, so an edit saved within a second of the + previous one carries the same stamp; ``when`` forces a distinct moment. + """ + return datetime.now(UTC).replace(microsecond=0) + + +def resolve_when(when: datetime | None) -> datetime: + """The moment a stamping save writes: ``when`` normalized, or the clock. + + An explicit value is converted to UTC and truncated to the second, so it + lands in the same form :func:`default_now` produces instead of carrying an + offset or fractional seconds into the output. + + A naive value is refused rather than guessed at: reading it as UTC and + reading it as local time give moments hours apart, and picking one silently + writes a date the caller did not mean. + """ + if when is None: + return default_now() + if when.tzinfo is None or when.tzinfo.utcoffset(when) is None: + raise ValueError( + f"when must be timezone-aware, got {when!r}: pass a UTC moment, " + "e.g. datetime.now(timezone.utc)" + ) + return when.astimezone(UTC).replace(microsecond=0) + + +def _dates_differ(left: datetime | date | None, right: datetime | date | None) -> bool: + """Whether two dates would reach the document as different attribute values. + + Rendered rather than compared as moments: re-expressing a date in another + offset is the same instant and equal to ``==``, but it changes the bytes, + and it is the one thing the caller touched. + """ + return _fmt_opt_date(left) != _fmt_opt_date(right) + + +def _needs_stamp(entry: Entry, baseline: _EntryRecord | None, digest: bytes) -> bool: + """Whether the content moved since ``baseline`` while the date stayed put. + + Both halves matter. The digest covers the entry's whole subtree, so it + catches an edit at any depth; the date comparison is what leaves an entry + the caller dated by hand alone, since a deliberate value is not one to + overwrite. + + Without a baseline there is nothing to compare — an entry added since the + load, or a lexicon built from scratch. A date already on such an entry is + taken as deliberate too (a migration carrying dates in from another format + is the case that matters), so only a blank one is filled. + """ + if baseline is None: + return entry.date_modified is None + return digest != baseline.digest and not _dates_differ( + entry.date_modified, baseline.date_modified + ) + + +@dataclass(slots=True) +class _StampUndo: + """How to put a stamping pass back if the write it ran for never happens. + + Stamping precedes serialization, so without this anything that kept the + ``.lift`` write from landing would leave the model dated for output that + does not exist. + """ + + lexicon: Lexicon + stamps: dict[int, _EntryRecord] # the baseline dict the pass replaced + dates: list[tuple[Entry, datetime | date | None, datetime | date | None]] + + def restore(self) -> None: + for entry, created, modified in self.dates: + entry.date_created = created + entry.date_modified = modified + self.lexicon._stamps = self.stamps + + +def note_caller_dates(lexicon: Lexicon) -> _StampUndo: + """Adopt the dates a ``stamp=False`` save writes as the new stamping baseline. + + Such a save writes what the model holds, so a date the caller put on an + entry is the date now on disk. The next stamping save has to measure a + further edit against that, not against the load. Without this step + :func:`_needs_stamp` would go on seeing a date that differs from the loaded + one, read it as deliberate, and never stamp the entry again. + + Only an entry whose date moved off its baseline is noted, and only that + entry is digested. An entry written unstamped under the date it was loaded + with is left out on purpose: its content is now on disk under a date that + no longer describes it, and the next stamping save should still say so. + + The rest of the baseline carries over from the last save, less any entry + that has since left the lexicon — dropped for the same reason + :func:`stamp_entries` rebuilds its own dict each pass. That function adopts + the dates it writes the same way. + """ + at_parse = _parse_time_records(lexicon) + previous = lexicon._stamps + entries = _by_identity(lexicon) + stamps = {id(entry): previous[id(entry)] for entry in entries if id(entry) in previous} + for entry in entries: + key = id(entry) + baseline = previous.get(key) + if baseline is None: + baseline = at_parse.get(key) + if baseline is None: + if entry.date_modified is None: + continue # an undated new entry is still new + elif not _dates_differ(entry.date_modified, baseline.date_modified): + continue + stamps[key] = _EntryRecord(entry, entry_digest(entry), entry.date_modified) + lexicon._stamps = stamps + return _StampUndo(lexicon, previous, []) + + +def _parse_time_records(lexicon: Lexicon) -> dict[int, _EntryRecord]: + source = lexicon._source + if source is None: + return {} + return {id(record.entry): record for record in source.entry_records} + + +def _by_identity(lexicon: Lexicon) -> list[Entry]: + """The entries, each once: one object holds one pair of dates.""" + return list({id(entry): entry for entry in lexicon.entries}.values()) + + +def stamp_entries(lexicon: Lexicon, when: datetime) -> _StampUndo: + """Stamp every entry whose content changed without its ``dateModified`` changing. + + Costs one canonical serialization pass over the entries. Returns what undoes + it, for a caller whose write does not go through. + + Each save leaves behind the state it wrote, in ``lexicon._stamps``, and the + next save measures against that rather than against the load. Without it a + second round of edits on the same in-memory lexicon would ship unstamped. + Its content differs from the loaded content, but so does its date: the + first save's own stamp, which reads exactly like a date the caller set + deliberately. An entry still matching its parse-time record needs no such + override and is recorded nowhere, so a save of a few edited entries + remembers only those. Rebuilding the dict each pass also drops entries that + have since left the lexicon, which would otherwise stay alive here for a + baseline nothing will consult again. + + Deciding comes before mutating because digesting is the only step that can + fail (see :func:`_guarded`). Content XML cannot represent is refused with + nothing stamped, rather than half-stamped at the entry it was found on. + """ + at_parse = _parse_time_records(lexicon) + previous = lexicon._stamps + + planned: list[tuple[Entry, bytes, bool]] = [] + for entry in _by_identity(lexicon): + baseline = previous.get(id(entry)) + if baseline is None: + baseline = at_parse.get(id(entry)) + digest = entry_digest(entry) + planned.append((entry, digest, _needs_stamp(entry, baseline, digest))) + + dates: list[tuple[Entry, datetime | date | None, datetime | date | None]] = [] + stamps: dict[int, _EntryRecord] = {} + for entry, digest, needs_stamp in planned: + if needs_stamp: + dates.append((entry, entry.date_created, entry.date_modified)) + entry._stamp(when) + digest = entry_digest(entry) # the dates are part of an entry's bytes + record = at_parse.get(id(entry)) + if ( + record is None + or record.digest != digest + or _dates_differ(record.date_modified, entry.date_modified) + ): + stamps[id(entry)] = _EntryRecord(entry, digest, entry.date_modified) + lexicon._stamps = stamps + return _StampUndo(lexicon, previous, dates) + + # --- canonical building blocks --------------------------------------------------- diff --git a/tests/test_stamp.py b/tests/test_stamp.py new file mode 100644 index 0000000..7114052 --- /dev/null +++ b/tests/test_stamp.py @@ -0,0 +1,669 @@ +"""Generated ``dateModified``/``dateCreated`` on save. + +The unit is the entry: across the seven FieldWorks 8.3-9.0 exports in The +Combine's ``Backend.Tests/Assets``, all 35,318 entries carry both stamps and not +one of 69,754 sub-entry nodes carries either. An entry's digest already covers +its whole subtree, so an edit at any depth belongs to the entry containing it. +""" + +import zipfile +from datetime import UTC, datetime, timedelta, timezone +from pathlib import Path + +import pytest + +import sil_lift + +CORPUS_DIR = Path(__file__).parent / "corpus" +DATED = CORPUS_DIR / "misc" / "sample.0.13.lift" # every entry carries both stamps +UNDATED = CORPUS_DIR / "spec-examples" / "0.13" / "subsenses.lift" # no entry carries either + +# Dates no typed field can hold: the reader keeps them as residue instead. +UNPARSEABLE_DATES = b""" + + +one + + +""" + +WHEN = datetime(2026, 3, 4, 5, 6, 7, tzinfo=UTC) +LATER = datetime(2026, 3, 4, 5, 6, 8, tzinfo=UTC) +BY_HAND = datetime(1999, 12, 31, 23, 59, 59, tzinfo=UTC) + + +def _dates(lexicon: sil_lift.Lexicon) -> dict[str | None, tuple[object, object]]: + """Every entry's stamps, keyed by id so a sort() does not disturb the comparison.""" + return {entry.id: (entry.date_created, entry.date_modified) for entry in lexicon.entries} + + +def _in_its_own_folder(fixture: Path, tmp_path: Path) -> Path: + """A copy of a fixture on its own: save_zip packages its whole folder.""" + folder = tmp_path / "src" + folder.mkdir() + dest = folder / fixture.name + dest.write_bytes(fixture.read_bytes()) + return dest + + +def test_an_untouched_save_stamps_nothing_and_stays_byte_identical(tmp_path: Path) -> None: + """Stamping is driven by content, so a load-and-save writes the source bytes back.""" + lexicon = sil_lift.load(DATED) + before = _dates(lexicon) + out = tmp_path / "out.lift" + lexicon.save(out, when=WHEN) + + assert _dates(lexicon) == before + assert out.read_bytes() == DATED.read_bytes() + # Every entry still matches its parse-time record, so nothing had to be + # remembered for the next save — the bookkeeping tracks edits, not entries. + assert lexicon._stamps == {} + + +def test_an_edit_at_any_depth_stamps_the_containing_entry(tmp_path: Path) -> None: + lexicon = sil_lift.load(UNDATED) + entry = lexicon.entries[0] + entry.senses[0].subsenses[0].glosses[0].text = sil_lift.Text(["edited"]) + lexicon.save(tmp_path / "out.lift", when=WHEN) + + assert entry.date_modified == WHEN + assert entry.date_created == WHEN # blank before, so filled with the same moment + + +def test_only_the_edited_entry_is_stamped(tmp_path: Path) -> None: + lexicon = sil_lift.load(DATED) + assert len(lexicon.entries) > 1 + target = lexicon.entries[3] + untouched = [entry for entry in lexicon.entries if entry is not target] + before = [(entry.date_created, entry.date_modified) for entry in untouched] + target.lexical_unit["en"] = "edited" + lexicon.save(tmp_path / "out.lift", when=WHEN) + + assert target.date_modified == WHEN + assert [(entry.date_created, entry.date_modified) for entry in untouched] == before + + +def test_an_existing_date_created_survives_the_stamp(tmp_path: Path) -> None: + """dateCreated is filled only when blank: an edit does not re-create an entry.""" + lexicon = sil_lift.load(DATED) + target = lexicon.entries[0] + created = target.date_created + assert created is not None + target.lexical_unit["en"] = "edited" + lexicon.save(tmp_path / "out.lift", when=WHEN) + + assert target.date_created == created + assert target.date_modified == WHEN + + +def test_stamp_false_writes_the_model_exactly_as_it_stands(tmp_path: Path) -> None: + lexicon = sil_lift.load(DATED) + target = lexicon.entries[0] + before = target.date_modified + target.lexical_unit["en"] = "edited" + lexicon.save(tmp_path / "out.lift", stamp=False) + + assert target.date_modified == before + assert lexicon._stamps == {} # a save that stamps nothing remembers nothing + + +def test_sorting_alone_stamps_nothing(tmp_path: Path) -> None: + """Matches the guarantee on Lexicon.sort: reordering leaves entry bytes alone.""" + lexicon = sil_lift.load(DATED) + before = _dates(lexicon) + lexicon.sort() + lexicon.save(tmp_path / "out.lift", when=WHEN) + + assert _dates(lexicon) == before + + +def test_a_second_round_of_edits_on_the_same_lexicon_is_also_stamped(tmp_path: Path) -> None: + """The baseline moves with each save; without that, only the first edit would bump. + + After the first save the entry's date differs from the loaded one — this + library's own stamp — which is indistinguishable from a caller-set date + unless the save records what it wrote. + """ + lexicon = sil_lift.load(DATED) + entry = lexicon.entries[0] + out = tmp_path / "out.lift" + + entry.lexical_unit["en"] = "first" + lexicon.save(out, when=WHEN) + assert entry.date_modified == WHEN + + entry.lexical_unit["en"] = "second" + lexicon.save(out, when=LATER) + assert entry.date_modified == LATER + + +def test_a_save_with_no_intervening_edits_leaves_the_stamp_alone(tmp_path: Path) -> None: + """Stamping twice for one edit would make every save a change downstream.""" + lexicon = sil_lift.load(DATED) + entry = lexicon.entries[0] + entry.lexical_unit["en"] = "edited" + first = tmp_path / "first.lift" + lexicon.save(first, when=WHEN) + + second = tmp_path / "second.lift" + lexicon.save(second, when=LATER) + + assert entry.date_modified == WHEN + assert second.read_bytes() == first.read_bytes() + + +def test_a_hand_set_date_becomes_the_baseline_for_the_next_edit(tmp_path: Path) -> None: + """Deliberate for the save it was set for, stale once the content moves again.""" + lexicon = sil_lift.load(DATED) + entry = lexicon.entries[0] + out = tmp_path / "out.lift" + + entry.lexical_unit["en"] = "edited" + entry.date_modified = BY_HAND + lexicon.save(out, when=WHEN) + assert entry.date_modified == BY_HAND + + entry.lexical_unit["en"] = "edited again" + lexicon.save(out, when=LATER) + assert entry.date_modified == LATER + + +def test_an_unstamped_save_notes_a_date_the_caller_set(tmp_path: Path) -> None: + """Which kind of save ran in between must not decide whether a later edit is stamped. + + An unstamped save writes the caller's date to disk, so it becomes the date + the next stamping save measures against — as it would have if that save had + stamped. Otherwise the entry reads as deliberately dated for good and is + never stamped again. + """ + lexicon = sil_lift.load(DATED) + entry = lexicon.entries[0] + out = tmp_path / "out.lift" + + entry.lexical_unit["en"] = "one" + entry.date_modified = BY_HAND + lexicon.save(out, stamp=False) + assert entry.date_modified == BY_HAND # written as it stands, as asked + + entry.lexical_unit["en"] = "two" + lexicon.save(out, when=WHEN) + assert entry.date_modified == WHEN + + +def test_an_unstamped_save_leaves_an_untouched_date_to_the_next_stamp( + tmp_path: Path, +) -> None: + """The other half: content on disk under a date that no longer describes it. + + Nothing is noted for an entry written unstamped under the date it was + loaded with, so the next stamping save still has the edit to report. + """ + lexicon = sil_lift.load(DATED) + entry = lexicon.entries[0] + loaded = entry.date_modified + out = tmp_path / "out.lift" + + entry.lexical_unit["en"] = "edited" + lexicon.save(out, stamp=False) + assert entry.date_modified == loaded + + lexicon.save(out, when=WHEN) # no further edit needed: the first one still stands + assert entry.date_modified == WHEN + + +def test_the_same_instant_in_another_offset_is_the_caller_touching_the_date( + tmp_path: Path, +) -> None: + """Dates are compared as the document will carry them, not as moments. + + Rewriting `2008-12-12T09:42:48+10:00` as the same instant at `-05:00` is an + edit — the file changes — and it is an edit to the date itself, so it is the + one thing a stamp must not overwrite. Comparing the two as moments would + call them equal and take the entry for content edited without its date. + """ + lexicon = sil_lift.load(DATED) + entry = lexicon.entries[0] + loaded = entry.date_modified + assert isinstance(loaded, datetime) + restated = loaded.astimezone(timezone(timedelta(hours=-5))) + assert restated == loaded # the same moment to ==, a different attribute value + entry.date_modified = restated + + lexicon.save(tmp_path / "out.lift", when=WHEN) + + assert entry.date_modified == restated + assert entry.date_modified.utcoffset() == timedelta(hours=-5) + + +def test_clearing_a_loaded_date_is_left_alone(tmp_path: Path) -> None: + """Blanking the date is a deliberate value like any other, so no stamp replaces it. + + It is the only way to hand an edited entry to a consumer that wants the + attribute absent, and nothing distinguishes it from the hand-set date + above: both are the caller saying what the output should carry. + """ + lexicon = sil_lift.load(DATED) + entry = lexicon.entries[0] + entry.lexical_unit["en"] = "edited" + entry.date_modified = None + out = tmp_path / "out.lift" + lexicon.save(out, when=WHEN) + + assert entry.date_modified is None + assert _dates(sil_lift.load(out))[entry.id] == (entry.date_created, None) + + +def test_clearing_a_date_without_an_edit_is_left_alone_too(tmp_path: Path) -> None: + """A date is part of an entry's bytes, so clearing one moves the digest by itself. + + What holds the stamp off is the date having left its baseline, not the + digest standing still — which it does not here. + """ + lexicon = sil_lift.load(DATED) + entry = lexicon.entries[0] + entry.date_modified = None + out = tmp_path / "out.lift" + lexicon.save(out, when=WHEN) + + assert entry.date_modified is None + assert _dates(sil_lift.load(out))[entry.id] == (entry.date_created, None) + + +def test_clearing_a_date_an_entry_never_had_does_not_hold_off_the_stamp( + tmp_path: Path, +) -> None: + """``None`` over ``None`` is the value already there, not a date the caller moved. + + An entry read without a ``dateModified`` cannot tell a caller who cleared + it from one who never touched the field, so an edit stamps it as any other + edit does. + """ + lexicon = sil_lift.load(UNDATED) + entry = lexicon.entries[0] + entry.senses[0].subsenses[0].glosses[0].text = sil_lift.Text(["edited"]) + entry.date_modified = None + lexicon.save(tmp_path / "out.lift", when=WHEN) + + assert entry.date_modified == WHEN + + +def test_an_entry_added_after_load_is_stamped_only_when_its_date_is_blank( + tmp_path: Path, +) -> None: + """A new entry has nothing to compare against, so a date on it is taken as meant.""" + lexicon = sil_lift.load(UNDATED) + blank = sil_lift.Entry(id="blank") + carried = sil_lift.Entry(id="carried", date_created=BY_HAND, date_modified=BY_HAND) + lexicon.entries += [blank, carried] + out = tmp_path / "out.lift" + lexicon.save(out, when=WHEN) + + assert (blank.date_created, blank.date_modified) == (WHEN, WHEN) + assert (carried.date_created, carried.date_modified) == (BY_HAND, BY_HAND) + + # Its first save recorded a baseline, so the carried date is not frozen for good. + carried.lexical_unit["en"] = "edited" + lexicon.save(out, when=LATER) + assert carried.date_modified == LATER + assert carried.date_created == BY_HAND + + +def test_a_from_scratch_lexicon_is_stamped_on_its_first_save(tmp_path: Path) -> None: + """The build-an-export case: no baseline anywhere, and every entry is new.""" + entry = sil_lift.Entry(id="kanga") + entry.lexical_unit["seh"] = "nkhuku" + lexicon = sil_lift.Lexicon(entries=[entry]) + out = tmp_path / "out.lift" + lexicon.save(out, when=WHEN) + + assert (entry.date_created, entry.date_modified) == (WHEN, WHEN) + + # Re-saving an unedited from-scratch lexicon is not a modification either. + lexicon.save(out, when=LATER) + assert entry.date_modified == WHEN + + entry.lexical_unit["seh"] = "nkhukhu" + lexicon.save(out, when=LATER) + assert entry.date_modified == LATER + assert entry.date_created == WHEN + + +def test_stamping_does_not_reach_below_the_entry(tmp_path: Path) -> None: + """Entry-level only: no sub-entry node is stamped, whatever moved inside it.""" + lexicon = sil_lift.load(UNDATED) + entry = lexicon.entries[0] + sense = entry.senses[0] + sense.date_modified = BY_HAND + sense.subsenses[0].glosses[0].text = sil_lift.Text(["edited"]) + lexicon.save(tmp_path / "out.lift", when=WHEN) + + assert entry.date_modified == WHEN + assert sense.date_modified == BY_HAND + assert sense.date_created is None + assert sense.subsenses[0].date_modified is None + + +def test_an_unscannable_document_stamps_only_what_changed(tmp_path: Path) -> None: + """No byte snapshot, but the digests still date the edit and nothing else. + + Note what `changed_entries()` says here by contrast: every entry, because + `save()` does re-serialize the whole file. Stamping asks the narrower + question — what the caller modified — and answers it exactly. + """ + text = UNDATED.read_text(encoding="utf-8").replace('encoding="UTF-8"', 'encoding="UTF-16"') + source = tmp_path / "utf16.lift" + source.write_bytes(text.encode("utf-16")) + lexicon = sil_lift.load(source) + assert lexicon._source is None # no byte baseline, only a stamping one + entry = lexicon.entries[0] + out = tmp_path / "out.lift" + + lexicon.save(out, when=WHEN) + assert (entry.date_created, entry.date_modified) == (None, None) + assert len(lexicon.changed_entries()) == len(lexicon.entries) + + entry.senses[0].subsenses[0].glosses[0].text = sil_lift.Text(["edited"]) + lexicon.save(out, when=WHEN) + assert (entry.date_created, entry.date_modified) == (WHEN, WHEN) + + +def test_when_is_normalized_to_utc_at_seconds_precision(tmp_path: Path) -> None: + """Whatever shape the caller's moment is in, the output keeps the one form.""" + lexicon = sil_lift.load(UNDATED) + entry = lexicon.entries[0] + entry.lexical_unit["en"] = "edited" + out = tmp_path / "out.lift" + lexicon.save(out, when=datetime(2026, 3, 4, 5, 6, 7, 500000, tzinfo=UTC)) + + assert entry.date_modified == WHEN # the fraction is dropped, not rounded + assert b'dateModified="2026-03-04T05:06:07Z"' in out.read_bytes() + + entry.lexical_unit["en"] = "edited again" + offset = timezone(timedelta(hours=5, minutes=30)) + lexicon.save(out, when=datetime(2026, 3, 4, 10, 36, 8, tzinfo=offset)) + + assert entry.date_modified == LATER # the same moment, said in UTC + assert b'dateModified="2026-03-04T05:06:08Z"' in out.read_bytes() + + +def test_a_naive_when_is_refused_and_nothing_is_written(tmp_path: Path) -> None: + """UTC or local would put the stamp hours apart, so neither is assumed.""" + lexicon = sil_lift.load(UNDATED) + entry = lexicon.entries[0] + entry.lexical_unit["en"] = "edited" + out = tmp_path / "out.lift" + + with pytest.raises(ValueError, match="timezone-aware"): + lexicon.save(out, when=datetime(2026, 3, 4, 5, 6, 7)) + + assert entry.date_modified is None + assert not out.exists() + + +def test_a_refused_write_leaves_nothing_stamped(tmp_path: Path) -> None: + """Stamping commits with the write: no file, no modification date. + + The refusal comes from the last entry, so this also pins that the entries + ahead of it are not left half-stamped — deciding happens before mutating. + """ + lexicon = sil_lift.load(DATED) + edited, offending = lexicon.entries[0], lexicon.entries[-1] + before = _dates(lexicon) + baseline = dict(lexicon._stamps) + edited.lexical_unit["en"] = "edited" + offending.senses[0].glosses.append(sil_lift.Form(lang="en", text=sil_lift.Text(["\ud800"]))) + out = tmp_path / "out.lift" + + with pytest.raises(sil_lift.LiftWriteError): + lexicon.save(out, when=WHEN) + + assert _dates(lexicon) == before + assert lexicon._stamps == baseline + assert not out.exists() + + +def test_a_refused_companion_write_leaves_the_lift_alone(tmp_path: Path) -> None: + """Every document is rendered before any is written, companions included. + + A companion is written after the .lift, so content refused while rendering + one would otherwise leave the .lift rewritten and stamped for a save that + raised. + """ + source = tmp_path / "doc.lift" + source.write_bytes(UNDATED.read_bytes()) + lexicon = sil_lift.load(source) + before = source.read_bytes() + baseline = dict(lexicon._stamps) + offending = sil_lift.Range(id="etymology") + offending.label["en"] = "\ud800" + ranges_file = sil_lift.RangesFile() + ranges_file.ranges.append(offending) + lexicon.add_ranges_file(ranges_file, href="doc.lift-ranges") + lexicon.entries[0].lexical_unit["en"] = "edited" + + with pytest.raises(sil_lift.LiftWriteError): + lexicon.save(when=WHEN) + + assert source.read_bytes() == before + assert not (tmp_path / "doc.lift-ranges").exists() + assert lexicon.entries[0].date_modified is None + assert lexicon._stamps == baseline + + +def test_a_refused_zip_write_leaves_nothing_stamped(tmp_path: Path) -> None: + lexicon = sil_lift.load(_in_its_own_folder(UNDATED, tmp_path)) + entry = lexicon.entries[0] + entry.lexical_unit["en"] = "edited" + entry.senses[0].glosses.append(sil_lift.Form(lang="en", text=sil_lift.Text(["\ud800"]))) + + with pytest.raises(sil_lift.LiftWriteError): + lexicon.save_zip(tmp_path / "pkg.zip", when=WHEN) + + assert entry.date_modified is None + + +def test_a_removed_entry_drops_out_of_the_stamping_baseline(tmp_path: Path) -> None: + """The baseline dict is rebuilt each save, so it holds no entry the lexicon lost. + + An entry the document was loaded with is retained by its parse-time record + either way — that is what makes `removed_entries()` work — but one appended + and then dropped would otherwise be kept alive here, subtree and all. + """ + lexicon = sil_lift.load(UNDATED) + appended = sil_lift.Entry(id="temporary") + lexicon.entries.append(appended) + out = tmp_path / "out.lift" + lexicon.save(out, when=WHEN) + assert [record.entry.id for record in lexicon._stamps.values()] == ["temporary"] + + lexicon.entries.remove(appended) + lexicon.save(out, when=LATER) + + assert lexicon._stamps == {} + + +def test_an_unstamped_save_drops_a_removed_entry_too(tmp_path: Path) -> None: + """A stamp=False save adopts caller-set dates, carrying nothing the lexicon lost. + + It starts from the records the last save left rather than rebuilding, so + without a filter a dropped entry would stay alive here, subtree and all, + until the next stamping save. + """ + lexicon = sil_lift.load(UNDATED) + appended = sil_lift.Entry(id="temporary") + lexicon.entries.append(appended) + out = tmp_path / "out.lift" + lexicon.save(out, when=WHEN) + assert [record.entry.id for record in lexicon._stamps.values()] == ["temporary"] + + lexicon.entries.remove(appended) + lexicon.save(out, stamp=False) + + assert lexicon._stamps == {} + + +def test_a_re_added_entry_keeps_the_date_it_left_with(tmp_path: Path) -> None: + """The cost of dropping what left: a re-added entry reads as deliberately dated. + + The record that goes when an entry leaves is the only evidence that its + date is one this library generated, so the edit below ships under the old + stamp rather than a fresh one. + """ + lexicon = sil_lift.load(UNDATED) + entry = sil_lift.Entry(id="temporary") + lexicon.entries.append(entry) + out = tmp_path / "out.lift" + lexicon.save(out, when=WHEN) + assert entry.date_modified == WHEN + + lexicon.entries.remove(entry) + lexicon.save(out, when=WHEN) # the rebuild drops its record + lexicon.entries.append(entry) + entry.lexical_unit["en"] = "edited" + lexicon.save(out, when=LATER) + + assert entry.date_modified == WHEN # LATER, if #45 is taken up + + +def test_an_aliased_entry_is_stamped_once(tmp_path: Path) -> None: + """One entry object, one pair of dates — however many list slots point at it.""" + lexicon = sil_lift.load(UNDATED) + entry = lexicon.entries[0] + lexicon.entries.append(entry) # the same object, written out twice + entry.lexical_unit["en"] = "edited" + out = tmp_path / "out.lift" + lexicon.save(out, when=WHEN) + + assert entry.date_modified == WHEN + assert len(lexicon._stamps) == 1 + assert out.read_bytes().count(b'dateModified="2026-03-04T05:06:07Z"') == 2 + + +def test_two_saves_of_one_moment_share_a_stamp(tmp_path: Path) -> None: + """The documented limit of seconds precision, said with an explicit moment. + + The same second cannot hold two distinct dates, so a second edit saved + within one reads as no change to anything comparing dates. The stamping + baseline still tracks it, so the next distinct moment stamps normally. + """ + lexicon = sil_lift.load(DATED) + entry = lexicon.entries[0] + out = tmp_path / "out.lift" + + entry.lexical_unit["en"] = "one" + lexicon.save(out, when=WHEN) + entry.lexical_unit["en"] = "two" + lexicon.save(out, when=WHEN) + assert entry.date_modified == WHEN + + entry.lexical_unit["en"] = "three" + lexicon.save(out, when=LATER) + assert entry.date_modified == LATER + + +def test_the_default_clock_is_utc_at_seconds_precision(tmp_path: Path) -> None: + """No `when`: the wall clock, in the 20-character form real exports use.""" + lexicon = sil_lift.load(UNDATED) + entry = lexicon.entries[0] + entry.lexical_unit["en"] = "edited" + before = datetime.now(UTC).replace(microsecond=0) + out = tmp_path / "out.lift" + lexicon.save(out) + after = datetime.now(UTC) + + stamped = entry.date_modified + assert isinstance(stamped, datetime) + assert stamped.utcoffset() == timedelta(0) + assert stamped.microsecond == 0 + assert before <= stamped <= after + rendered = f'dateModified="{stamped.strftime("%Y-%m-%dT%H:%M:%SZ")}"'.encode() + assert rendered in out.read_bytes() + + +def test_a_stamped_save_is_byte_reproducible_given_when(tmp_path: Path) -> None: + """`when` is what keeps a stamping pipeline diffable: same input, same bytes.""" + outputs = [] + for name in ("first.lift", "second.lift"): + lexicon = sil_lift.load(UNDATED) + lexicon.entries[0].senses[0].subsenses[0].glosses[0].text = sil_lift.Text(["edited"]) + out = tmp_path / name + lexicon.save(out, when=WHEN) + outputs.append(out.read_bytes()) + + assert outputs[0] == outputs[1] + + +def test_validation_does_not_stamp(tmp_path: Path) -> None: + """iter_problems is read-only, so the bytes it reports on precede any stamp.""" + lexicon = sil_lift.load(DATED) + entry = lexicon.entries[0] + before = entry.date_modified + entry.lexical_unit["en"] = "edited" + + assert [problem for problem in lexicon.iter_problems() if problem.level == "error"] == [] + assert entry.date_modified == before + + lexicon.save(tmp_path / "out.lift", when=WHEN) + assert entry.date_modified == WHEN + + +def test_changed_entries_still_reports_a_stamped_entry(tmp_path: Path) -> None: + """The stamping baseline is its own: change detection still answers "since load".""" + lexicon = sil_lift.load(DATED) + entry = lexicon.entries[0] + entry.lexical_unit["en"] = "edited" + lexicon.save(tmp_path / "out.lift", when=WHEN) + + assert [id(reported) for reported in lexicon.changed_entries()] == [id(entry)] + assert lexicon.changes() + + +def test_stamping_replaces_a_date_the_model_could_not_hold(tmp_path: Path) -> None: + """A generated stamp wins over an unparseable date, which is only residue. + + A model field always beats stale residue in the writer, and that is the + right way round here: the entry is being rewritten, and "whenever" is no + date at all, so an edit leaves it with one that is. `stamp=False` is what + preserves the original strings. + """ + source = tmp_path / "junk.lift" + source.write_bytes(UNPARSEABLE_DATES) + lexicon = sil_lift.load(source) + entry = lexicon.entries[0] + assert (entry.date_created, entry.date_modified) == (None, None) + + entry.lexical_unit["en"] = "edited" + out = tmp_path / "out.lift" + lexicon.save(out, when=WHEN) + written = out.read_bytes() + assert b'dateCreated="2026-03-04T05:06:07Z" dateModified="2026-03-04T05:06:07Z"' in written + assert b"whenever" not in written + + kept = sil_lift.load(source) + kept.entries[0].lexical_unit["en"] = "edited" + unstamped = tmp_path / "unstamped.lift" + kept.save(unstamped, stamp=False) + assert b'dateCreated="nope" dateModified="whenever"' in unstamped.read_bytes() + + +def test_save_zip_stamps_by_default(tmp_path: Path) -> None: + """A package is the hand-off to the tools that reconcile on dateModified.""" + lexicon = sil_lift.load(_in_its_own_folder(UNDATED, tmp_path)) + entry = lexicon.entries[0] + entry.senses[0].subsenses[0].glosses[0].text = sil_lift.Text(["edited"]) + dest = tmp_path / "pkg.zip" + lexicon.save_zip(dest, when=WHEN) + + assert entry.date_modified == WHEN + with zipfile.ZipFile(dest) as archive: + member = next(name for name in archive.namelist() if name.endswith(".lift")) + assert b'dateModified="2026-03-04T05:06:07Z"' in archive.read(member) + + +def test_save_zip_stamp_false_leaves_the_model_alone(tmp_path: Path) -> None: + lexicon = sil_lift.load(_in_its_own_folder(UNDATED, tmp_path)) + entry = lexicon.entries[0] + entry.senses[0].subsenses[0].glosses[0].text = sil_lift.Text(["edited"]) + lexicon.save_zip(tmp_path / "pkg.zip", stamp=False) + + assert entry.date_modified is None