Skip to content

Stamp dateModified on the entries save() rewrites - #43

Draft
imnasnainaec wants to merge 6 commits into
mainfrom
stamp-date-modified-on-save
Draft

imnasnainaec wants to merge 6 commits into
mainfrom
stamp-date-modified-on-save

Conversation

@imnasnainaec

@imnasnainaec imnasnainaec commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Resolves #10, taking Route 1: save() stamps the entries it rewrites, by default.

What changed

Lexicon.save() and Lexicon.save_zip() take two keyword-only parameters:

  • stamp: bool = True — set dateModified on every entry whose content moved while its date stayed where its baseline had it, and fill a blank dateCreated with the same moment. stamp=False writes the model exactly as it stands.
  • when: datetime | None = None — the moment to write, in place of the wall clock. Timezone-aware, normalized to UTC whole seconds, which is what makes stamped output byte-reproducible.

Entries only, as the survey in #10 concluded. An entry's digest spans its whole subtree, so an edit to a gloss on a nested subsense stamps the entry containing it. The policy lives in one private method, so the other eight date-bearing types inherit it if they ever need it.

Three cases are left as they stand: an entry the caller dated deliberately, an entry created since the load that already carries a date, and an untouched entry — reordering included.

Decisions #10 left open

Entries with no baseline — appended after the load, or in a from-scratch lexicon — are stamped only where dateModified is blank, rather than unconditionally as Route 1 words it.

Unconditional stamping would overwrite dates an exporter carried in from its own data model, and would rewrite every date on every save of a from-scratch lexicon. Such an entry's first save records a baseline either way, so a later edit to a date-carrying one does bump.

iter_problems() stays read-only. Its docstring no longer claims to validate "what save() would write": it says the bytes it reports on precede the stamping a save does, and that nothing generated is ever a finding, so validate-then-save is sound.

save_zip() stamps by default too. A package is the hand-off to the tools that reconcile on dateModified, so leaving it unstamped would reproduce the bug in the case that matters most.

Baseline bookkeeping

The stamping baseline is deliberately not the parse-time one.

  • Each save records what it wrote in Lexicon._stamps, and the next save measures against that. Without it, a second round of edits on one loaded lexicon would ship unstamped: its date differs from the load, which is exactly what a caller-set date looks like.
  • The parse-time records keep driving byte reuse and change detection, untouched. Overwriting them on stamp would make changed_entries() stop reporting a stamped entry after a save, contradicting its documented "always against the load, never against the most recent save()" guarantee.
  • A save records only entries that deviate from their parse-time record, and only those still in the lexicon. The bookkeeping is the size of the edit, and holds nothing the lexicon has dropped.

_EntryRecord gains the dateModified it held when the record was taken. That is what separates "the content changed and the date did not" from "the caller set the date deliberately".

One fidelity consequence worth a look

An unparseable date (dateModified="whenever") is residue, not a date, so the model field is None. An edited entry therefore gets stamped, replacing the original string — which a stamp=False save would have preserved.

The alternative, treating unparseable-as-present and skipping the stamp, would leave the worst data the only data that never gets a usable date. Pinned by a test and documented in docs/en/fidelity.md.

Write ordering

save() renders the .lift and every companion before writing any of them. Content XML cannot represent is then refused with nothing on disk and nothing stamped, wherever in the set it sits. save_zip() already staged its whole package this way.

Cost

One extra canonical serialization pass over the entries per stamping save; Route 1's table claimed none.

Sharing digests with render_document would mean threading a cache through it and through _validate/_zip, which seemed worse than documenting the pass the way changed_entries() and changes() already document theirs.

Not in scope

SOURCE_DATE_EPOCH as the default clock source (#9). default_now() is the single place the clock is read, so that stays a one-function change on top. when= already makes the byte-exact tests here writable without monkeypatching.

Tests and docs

tests/test_stamp.py is new: the rule at depth, the three left-alone cases, the baseline across successive saves, byte-reproducibility under when=, both save_zip paths, and the residue case above. The corpus byte-identity and Hypothesis round-trip suites pass unchanged, which is the point — stamping is driven by content, so an unedited document is stamped nowhere.

Docs: docs/en/fidelity.md, the editing and build-an-export guides, validate.md, lift-export-interop.md, large-files.md (the streaming writer stamps nothing, and why), read-edit-write.md, index.md, README.md, and the [0.1.0] changelog section.

🤖 Generated with Claude Code


This change is Reviewable

@imnasnainaec

This comment was marked as outdated.

@imnasnainaec

This comment was marked as outdated.

imnasnainaec and others added 4 commits September 14, 2026 08:48
Lexicon.save() and save_zip() take stamp (default True) and when. Before
serializing, stamp_entries() sets dateModified — filling a blank dateCreated
with the same moment — on every entry whose canonical digest moved while its
date stayed where its baseline had it. Until now nothing in the library
generated a timestamp, so an edited entry went out under the date it was read
with and every tool that reconciles LIFT on dateModified saw an unmodified
lexicon. stamp=False writes the model exactly as it stands; when= supplies the
moment in place of the wall clock (UTC at seconds precision, the 20-character
form real FieldWorks exports use without exception), which is what makes
stamped output byte-reproducible.

Entries only. All 35,318 entries in the seven FieldWorks 8.3-9.0 exports in The
Combine's Backend.Tests/Assets carry both stamps and not one of their 69,754
sub-entry nodes carries either, and an entry's digest already spans its whole
subtree, so an edit to a nested subsense stamps the entry containing it.
_ExtensibleNoFields._stamp holds the policy, so the other eight date-bearing
types inherit it if they ever need it.

_EntryRecord gains the dateModified it held when the record was taken, which is
what separates "the content changed and the date did not" from "the caller set
the date deliberately". The parse-time records keep driving byte reuse and
change detection; each save records what it wrote in Lexicon._stamps and the
next save measures against that, without which a second round of edits on one
loaded lexicon would read as caller-set and ship unstamped. Keeping the two
baselines apart is what leaves changed_entries() answering "since the load"
rather than "since the last save". An entry still matching its parse-time
record is recorded nowhere, so the bookkeeping is the size of the edit.

An entry with no baseline at all — appended after the load, or in a lexicon
built from scratch — is stamped only where its dateModified is blank, so an
exporter carrying real dates in from another data model keeps them. Its first
save records a baseline either way, so a later edit to it does bump.

iter_problems() does not stamp. It stays read-only, and its docstring says the
bytes it validates precede the stamping a save does rather than claiming to be
what save() would write; nothing generated is ever a finding, so validating
first and saving after is sound. canonicalize() generates nothing either:
sorting and reformatting change no entry's content.

An unparseable date is residue rather than a date, so a stamp replaces it and
the original string is dropped — a consequence pinned by a test and documented
alongside the rest in docs/en/fidelity.md, with the guides that teach editing
and building an export, and the streaming writer's note that it stamps nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five corrections to entry stamping.

A document the byte scanner declined had no stamping baseline at all, so
every undated entry read as new and a save that changed nothing dated all of
them. Byte reuse needs the source bytes but stamping needs only the digests,
so the reader now records digests and dates for such a document too, and a
no-op save of one leaves it alone.

when= is normalized to UTC at whole seconds, so an explicit moment reaches
the output in the one form the rest of it uses instead of carrying an offset
or fractional seconds through _fmt_date. A naive value is refused: read as UTC
and read as local time it names moments hours apart, and picking one silently
writes a date the caller did not mean.

Stamping now commits with the write. stamp_entries returns what undoes it,
save() and save_zip() put the dates and the baseline back when the write does
not go through, and the pass decides before it mutates so that the one step
that can fail — digesting content XML cannot represent — refuses with nothing
stamped rather than half-stamped at whatever entry the refusal came from.

The baseline dict is rebuilt each pass rather than updated in place, so an
entry appended and later removed is no longer held alive, with its whole
subtree, by a record nothing will consult again. Entries the document was
loaded with are retained by their parse-time records as before.

Entries are iterated by identity, so an entry aliased into the list twice is
decided and stamped once — it is one object with one pair of dates, whatever
its output happens to say twice.

default_now() documents what seconds precision costs: one second holds one
date, so a second edit saved inside the same second as the first carries the
same stamp. Sub-second precision would buy the distinction at the cost of the
form every consumer expects, and when= forces a distinct moment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Internal vocabulary is out of the guides: a reader has no use for the word
baseline, and the streaming note now says what the limitation is — a pass that
never saw the document cannot tell which entries changed — rather than naming
the machinery it lacks.

Refused-write behavior is stated where it is a contract, in fidelity.md and on
save(), and no longer repeated in a guide whose script aborts before it saves.
The comments on the clock, the undo, the identity keying and the two-phase pass
keep their reasons and drop the reassurance around them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two paths left a date the caller set unstampable for good.

A save that stamps nothing still writes what the model holds, so a date the
caller put there is the date now on disk. note_caller_dates records it, and the
next stamping save measures a further edit against it. Without that the entry
kept failing the stale test — its date differs from the load, which is exactly
what a deliberate date looks like — and was never stamped again, while the same
sequence through a stamping save bumped normally. An entry written unstamped
under the date it was loaded with is deliberately not noted: its content is on
disk under a date that no longer describes it, and the next stamping save
should still say so.

Dates are compared as the document will carry them rather than as moments. Two
aware values an hour and an offset apart are the same instant and equal to ==,
so restating 2008-12-12T09:42:48+10:00 at -05:00 — an edit, and an edit to the
date itself — read as content changed without its date, and the stamp
overwrote the one thing the caller had touched.

The parse-time record lookup and the identity keying move into helpers shared
by both passes, and _apply_stamps now always returns an undo.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@imnasnainaec
imnasnainaec force-pushed the stamp-date-modified-on-save branch from fadf3b3 to 908c35a Compare September 14, 2026 12:48
imnasnainaec and others added 2 commits September 14, 2026 09:39
Lexicon.save() wrote the .lift and then rendered each companion in turn, so
content XML cannot represent — a lone surrogate in a range label, say — was
refused only once the .lift was already on disk, rewritten and stamped for a
save that raised. The method's own contract says nothing is written and nothing
is stamped in that case. The .lift, its companions, and the paths they go to are
now all settled before the first byte lands, which is how save_zip has always
worked: it stages the whole package in a temp directory and writes the archive
once.

Companion targets move into _companion_targets(), since the write loop now runs
on rendered bytes and has nothing left to decide. Failure past the .lift write
is unchanged: the file on disk carries those stamps, so they stand.

This buys refusal, not atomicity. An OS error partway through the companions
still leaves a mix of new and old files on disk; that would want staged writes
and renames across every path that writes a document.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
note_caller_dates started from a copy of the whole previous baseline and only
ever added to it, so a record for an entry appended and later removed survived
every stamp=False save. _EntryRecord holds its Entry strongly, to keep that
object's id() from being reused while the record stands, which means such a
record also kept the removed entry's whole subtree alive until the next stamping
save rebuilt the dict from the entries present. Filtering the copy to those same
entries gives both passes one rule.

Dropping the record is also what erases the only evidence that the date an
appended entry carries was generated here rather than set deliberately, so an
entry that leaves, is saved past, and comes back keeps the date it left with
even after a further edit. That is the standing cost of holding nothing the
lexicon no longer holds, and a test pins it so it reads as a decision rather
than an oversight.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update dateModified when a lexicon is changed: two candidate designs

1 participant