Skip to content
22 changes: 9 additions & 13 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,20 +39,16 @@ releases may contain breaking changes.
represent — a lone surrogate, which only an API assignment can introduce — is
refused with `LiftWriteError` naming the node, and reported by validation as
`lone-surrogate`.
- Generated timestamps on save: `Lexicon.save()` and `Lexicon.save_zip()` stamp
`dateModified` on every entry whose content changed since it was loaded, and
fill a blank `dateCreated` with the same moment. `stamp=False` turns it off;
`when=` supplies the moment in place of the wall clock.
- Change detection against the loaded document, reading the same parse-time
digests. `Lexicon.changed_entries()` reports entries whose content differs
(an entry's digest covers its whole subtree, so an edit at any depth reports
the entry containing it, while an identical rewrite or a `sort()` reports
nothing); `added_entries()` and `removed_entries()` report the rest — an
entry already in the document, appended a second time, counts as an addition
— and need no serialization. `Lexicon.changes()` and `RangesFile.changes()`
return `Changes` / `RangesChanges`, covering entry content, additions,
removals, reordering, the header, the root element, and every tracked
companion — each falsy only when `save()` would reproduce the source bytes,
so `if not lex.changes():` is a correct guard for skipping an in-place write
(content, not destination: a `save(path)` into another directory writes
there regardless). Comparison is always against the loaded document, never
against the most recent `save()`.
digests. `Lexicon.changed_entries()`, `added_entries()`, and
`removed_entries()` report entries; `Lexicon.changes()` and
`RangesFile.changes()` return a `Changes` / `RangesChanges` summary covering
entry content, additions, removals, reordering, the header, the root
element, and tracked companions.
- LIFT-folder handling: `RangesFile` (standalone `.lift-ranges` documents,
same fidelity guarantees), automatic companion discovery/tracking on load
(`Lexicon.ranges_files`), `save()` writes companions together,
Expand Down
8 changes: 5 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,12 @@ canonical sorting — with streaming APIs for large lexicons.
```python
import sil_lift

lex = sil_lift.load("thesaurus.lift") # tracks .lift-ranges companions too
lex = sil_lift.load("thesaurus.lift") # tracks .lift-ranges companions too
for entry in lex.entries:
...
entry = lex.find(id="hoofd_a1b2")
entry.senses[0].definition["en"] = "head (anatomy)"
lex.save() # untouched entries byte-identical
lex.save() # edited entries stamped; rest verbatim
```

**Status: pre-release, under active development.** The API is not yet stable.
Expand All @@ -35,7 +35,9 @@ not understand.
- Entries you did not modify are written back **byte-identical**, even when
other entries changed.
- Entries you did modify are re-serialized in a documented canonical form, with
all out-of-schema content (unknown elements, attributes, comments) preserved.
all out-of-schema content (unknown elements, attributes, comments) preserved,
and are stamped with a fresh `dateModified` so the edit does not go out under
the date it was loaded with.
- Whitespace inside `<text>` is never altered — not even for indentation.

The precise rules and their few edge cases are documented in
Expand Down
24 changes: 22 additions & 2 deletions docs/en/fidelity.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Any well-formed LIFT 0.13 document loads — schema-invalid content included. Wh

## Saving an unchanged document

`load()` → `save()` with no edits writes **byte-identical output** — no reformatting, no re-escaping, no reordering, byte-order marks and XML declarations included. There is currently no normalization list: identity is exact.
`load()` → `save()` with no edits writes **byte-identical output** — no reformatting, no re-escaping, no reordering, byte-order marks and XML declarations included. There is currently no normalization list: identity is exact. Timestamps are generated from content, so an unchanged document has none generated either.

Exceptions (the writer falls back to full canonical serialization, which is semantically complete but not byte-preserving):

Expand All @@ -20,12 +20,32 @@ Exceptions (the writer falls back to full canonical serialization, which is sema
## Saving an edited document

- **Untouched entries are emitted verbatim from their original bytes.** An entry counts as touched if any part of its model object changed since parse (detected by canonical-serialization snapshot, not a dirty flag).
- **Touched entries are re-serialized canonically and completely**: UTF-8, 2-space indentation _outside_ mixed content (whitespace inside `<text>` and `<span>` is never altered), a documented child grouping per element (e.g. entry: lexical-unit, citation, pronunciations, variants, senses, notes, relations, etymologies, annotations, traits, fields), fixed attribute order, dates in ISO-8601 (`Z` for UTC). All residue is re-emitted; its position is restored to the original child index, clamped to the new child list (an approximation — exact byte positions are only guaranteed for untouched entries).
- **Touched entries are re-serialized canonically and completely.** Canonical form is:
- UTF-8.
- 2-space indentation _outside_ mixed content; whitespace inside `<text>` and `<span>` is never altered.
- A documented child grouping per element; for `<entry>`: lexical-unit, citation, pronunciations, variants, senses, notes, relations, etymologies, annotations, traits, fields.
- Fixed attribute order.
- Dates in ISO-8601 (`Z` for UTC).
- **All residue is re-emitted.** Its position is restored to the original child index, clamped to the new child list — an approximation, since exact byte positions are only guaranteed for untouched entries.
- Adding, removing, or reordering entries re-serializes the document structure but still emits every unchanged entry's bytes verbatim.
- **A touched entry is stamped** with a fresh `dateModified`, and a `dateCreated` if it had none — see [Generated timestamps](#generated-timestamps).

!!! note "&quot;Canonical&quot; here is not related to any other Canonical XML"
Canonical form on this page means `sil-lift`'s own documented shape, described in a bullet above. It is unrelated to W3C's Canonical XML (C14N) process. It is unrelated to `SIL.Core`'s `CanonicalXmlSettings` class.

## Generated timestamps

A generated stamp is the one thing in the output that is not a function of the input.

- **What is stamped.** Every entry whose content changed since it was read, with a fresh `dateModified` and, if it had none, a `dateCreated` of the same moment. An edit shipped under its loaded date looks unmodified to everything that reconciles on that attribute, FieldWorks and The Combine's LIFT import included.
- **Entries only.** No node below an `<entry>`, and nothing in the header.
- **What is left alone.** An entry whose date the caller set deliberately, and an entry created since the load that already carries one.
- **An unparseable date is replaced.** A date the model could not parse is [residue](#reading) rather than a date, so a stamp overwrites it and the original string is dropped — an edited entry is better off carrying a real date than `dateModified="whenever"`.
- **The moment.** UTC at seconds precision (`YYYY-MM-DDTHH:MM:SSZ`, the shape every surveyed FieldWorks export uses), read from the wall clock. One second holds one date, so an edit saved within a second of the previous one carries the same stamp.
- **`save(when=...)`** supplies the moment instead of the wall clock, which is what keeps stamped output reproducible for a diff-based CI gate. It must be timezone-aware, and is normalized to UTC whole seconds.
- **`save(stamp=False)`** writes the model exactly as it stands, residue included.
- **Stamping commits with the `.lift` write.** Anything that keeps that write from landing puts the dates back. Once it lands they stand, even if a companion write fails after it.

## Content XML cannot represent

Non-BMP characters — emoji, CJK Extension B, Adlam, anything above U+FFFF — are ordinary content and round-trip byte-identically. A "surrogate pair" is a UTF-16 encoding detail: Python strings are sequences of codepoints, so nothing in the reader, the byte scanner, or the writer ever sees one.
Expand Down
7 changes: 4 additions & 3 deletions docs/en/guides/build-export.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ ranges.add_range("grammatical-info").add_element("Noun").label["en"] = "noun"
ranges.add_range("semantic-domain-ddp4").add_element("1.6.1.2").label["en"] = "Bird"
lex.add_ranges_file(ranges, href="birds.lift-ranges")

# Validate what save() would write, before touching the disk.
# Validate the document as it stands, before touching the disk.
problems = list(lex.iter_problems())
print(f"validation: {len(problems)} problem(s)")

Expand All @@ -79,7 +79,7 @@ print((out / "birds.lift-ranges").read_text(encoding="utf-8"), end="")
<range id="semantic-domain-ddp4" href="birds.lift-ranges"/>
</ranges>
</header>
<entry id="kanga" guid="6b9e7c2a-3f4d-4a1b-8c5e-2d9f0a1b2c3d">
<entry id="kanga" guid="6b9e7c2a-3f4d-4a1b-8c5e-2d9f0a1b2c3d" dateCreated="2026-09-02T19:19:17Z" dateModified="2026-09-02T19:19:17Z">
<lexical-unit>
<form lang="seh">
<text>nkhuku</text>
Expand Down Expand Up @@ -161,7 +161,8 @@ print((out / "birds.lift-ranges").read_text(encoding="utf-8"), end="")
- A `URLRef` is an href plus an optional caption/label multitext — used for both `<media>` (audio) and `<illustration>` (photos). The pronunciation here follows The Combine's convention of an `en` form reading `Speaker: <name>`.
- App-specific data with no native LIFT home rides as a `<field>` (or `<trait>`): FieldWorks reads these as custom fields and The Combine preserves them.
- Give every entry a real, stable `guid` (e.g. from `uuid.uuid4()`, reused across exports) — a later re-import updates the entry in place rather than duplicating it. `sil-lift validate --require-ids` enforces this.
- `lex.iter_problems()` validates the in-memory document (what `save()` would write) before anything hits disk; here it is clean. Because the lexicon has no folder yet, the media-presence and companion-href checks are skipped — run [`sil-lift validate`](cli.md) on the saved output (or with `--no-check-media`) once the audio and photo files are in place.
- The `dateCreated`/`dateModified` in the output above are not in the script: `save()` stamped them with the moment it ran, since an entry written for the first time carries no date of its own. `when=` supplies that moment instead of reading the clock, which is what makes a generated export byte-reproducible for a CI job to diff; `stamp=False` writes no dates at all. See [Generated timestamps](../fidelity.md#generated-timestamps) for the rest of the contract.
- `lex.iter_problems()` validates the in-memory document before anything hits disk; here it is clean. Because the lexicon has no folder yet, the media-presence and companion-href checks are skipped — run [`sil-lift validate`](cli.md) on the saved output (or with `--no-check-media`) once the audio and photo files are in place.

## Packaging

Expand Down
1 change: 1 addition & 0 deletions docs/en/guides/bulk-edit-glosses.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ A few things worth noting:
- It compares content, not destination, so guard only an in-place save with it: `lex.save(some_other_dir / "dictionary.lift")` writes the document and its companions to a location that has nothing in it yet, whether or not anything changed.
- It is a guard, not a speed-up — answering it digests every entry, which is the same work `save()` does to decide which source bytes it can reuse, so what you skip is the write itself (an unchanged file-modification time, no spurious diff), not the effort of deciding.
- Validating in memory (`lex.iter_problems()`) serializes the edited state first, so it correctly reflects the edit before anything is written to disk. Aborting on any `"error"`-level `Problem` — warnings are left for the caller to decide about — means a bad edit never reaches `save()`.
- `lex.save()` stamps a fresh `dateModified` on exactly the entries this script changed, filling `dateCreated` where it was blank. Without it a lexicon this script has rewritten would look untouched to FieldWorks or to The Combine's LIFT import, both of which decide what to update from that attribute — and it stamps `<entry>` only, however deep the edit went: the entry containing the edited subsense gloss, not the sense.

Glosses aren't the only thing worth touching this way. The same `Multitext` mapping surface applies to definitions and every other multilingual field on an entry or sense:

Expand Down
1 change: 1 addition & 0 deletions docs/en/guides/large-files.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,5 @@ Notes:

- The writer's output is exactly what the full-document canonical serializer would produce for the same content — the two modes never drift apart.
- Streaming mode reuses no source bytes: output is always canonical. Root-level LIFT residue — comments between entries and out-of-schema attributes on `<lift>` — is not carried; entries and the header are complete, residue included.
- Nor does it generate timestamps, as [`Lexicon.save()`](../fidelity.md#saving-an-edited-document) does. An entry is written with the dates it carries: a streaming pass never sees the document as it was, so it cannot tell which entries you changed. Set `entry.date_modified` yourself on the ones this pass rewrites.
- If the body of an `open_writer` block raises, the file is left visibly unterminated (no closing `</lift>`) — a half-written lexicon must not look complete.
1 change: 1 addition & 0 deletions docs/en/guides/lift-export-interop.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ LIFT is usually moved around as a single `.zip` — FieldWorks and The Combine b
- Extraction is capped at 10 GiB and 100,000 members; a package over either limit is refused with a `LiftParseError`, as is one whose member paths escape the extraction directory.
- **Write:** `Lexicon.save_zip("out.zip", wrap_folder="MyDict")` packages the `.lift`, its `.lift-ranges`, and every other file in the source folder (media, `WritingSystems/`, `consent/`, ...) into a zip.
- `wrap_folder` defaults to a top-level folder named after the zip (the FieldWorks/Combine import convention); pass `False` for a flat archive.
- Entries whose content changed since the load are stamped with a fresh `dateModified` on the way out, exactly as on `save()`: a package is what an importing tool reconciles from, so a stale date there is what makes an updated lexicon look untouched. `stamp=False` and `when=` work the same way here.

The `.lift` and `.lift-ranges` keep their byte-fidelity inside the package; the zip container itself is not byte-reproducible.

Expand Down
2 changes: 2 additions & 0 deletions docs/en/guides/read-edit-write.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@ lex.save("elsewhere.lift")

Entries you didn't modify are written back **byte-identical**; a document you didn't modify at all is byte-identical from the first byte to the last. See [Fidelity guarantees](../fidelity.md) for the precise contract.

The entries you did modify go out with a fresh `dateModified` (and a `dateCreated` if they had none), so an edit doesn't ship under the date it was loaded with — the tools that merge LIFT decide what changed from that attribute. `lex.save(stamp=False)` writes the dates the model holds and nothing more; `lex.save(when=...)` pins the moment instead of reading the clock. See [Generated timestamps](../fidelity.md#generated-timestamps) for the rest of the rules.

## Building from scratch

```python
Expand Down
2 changes: 1 addition & 1 deletion docs/en/guides/validate.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Every finding carries one of these, whichever layer produced it — `schema` and
| `undefined-range-value` | warning | a grammatical-info or range-keyed trait value the range does not list |
| `uri-not-rfc` | warning | an href that is not a valid URI — FLEx's `file://C:/...` |

All three layers work from what `save()` would write, so a document that cannot be serialized at all is reported as a single `lone-surrogate` error instead — see [Fidelity guarantees](../fidelity.md#content-xml-cannot-represent).
All three layers work from the document serialized as it stands, so one that cannot be serialized at all is reported as a single `lone-surrogate` error instead — see [Fidelity guarantees](../fidelity.md#content-xml-cannot-represent). Validation is read-only: it reports the document as it stands, before the `dateModified` stamping a save does. Nothing generated is ever a finding, so validate-then-save is sound.

A companion name matching several files loads none of them: the ranges they define go absent until all but one is renamed or removed.

Expand Down
2 changes: 1 addition & 1 deletion docs/en/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,5 @@ for entry in lex.entries:
entry = lex.find(guid="0f5a9c3e-...") # or lex.find(id="hoofd_a1b2")
entry.senses[0].definition["en"] = "head (anatomy)"

lex.save() # untouched entries byte-identical; edited entry re-serialized
lex.save() # edited entry re-serialized and re-stamped; the rest byte-identical
```
4 changes: 4 additions & 0 deletions src/sil_lift/_canonical.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,10 @@ def canonicalize(src: str | os.PathLike[str], dst: str | os.PathLike[str]) -> No
diff cleanly. Text content is never whitespace-normalized. The whole
document is held in memory (sorting requires it; the C# oracle buffers too).

No timestamp is generated either: sorting and reformatting change no entry's
content, so nothing here is a modification to stamp. The output is a pure
function of the input.

Only the ``.lift`` file is written: companion ``.lift-ranges`` files are
neither read nor rewritten (the source is loaded with
``resolve_ranges=False``). Sort a ranges file separately via
Expand Down
Loading