Stamp dateModified on the entries save() rewrites - #43
Draft
imnasnainaec wants to merge 6 commits into
Draft
imnasnainaec wants to merge 6 commits into
imnasnainaec wants to merge 6 commits into
Conversation
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
Lexicon.save() and save_zip() take stamp (default True) and when. Before serializing, stamp_entries() sets dateModified — filling a blank dateCreated with the same moment — on every entry whose canonical digest moved while its date stayed where its baseline had it. Until now nothing in the library generated a timestamp, so an edited entry went out under the date it was read with and every tool that reconciles LIFT on dateModified saw an unmodified lexicon. stamp=False writes the model exactly as it stands; when= supplies the moment in place of the wall clock (UTC at seconds precision, the 20-character form real FieldWorks exports use without exception), which is what makes stamped output byte-reproducible. Entries only. All 35,318 entries in the seven FieldWorks 8.3-9.0 exports in The Combine's Backend.Tests/Assets carry both stamps and not one of their 69,754 sub-entry nodes carries either, and an entry's digest already spans its whole subtree, so an edit to a nested subsense stamps the entry containing it. _ExtensibleNoFields._stamp holds the policy, so the other eight date-bearing types inherit it if they ever need it. _EntryRecord gains the dateModified it held when the record was taken, which is what separates "the content changed and the date did not" from "the caller set the date deliberately". The parse-time records keep driving byte reuse and change detection; each save records what it wrote in Lexicon._stamps and the next save measures against that, without which a second round of edits on one loaded lexicon would read as caller-set and ship unstamped. Keeping the two baselines apart is what leaves changed_entries() answering "since the load" rather than "since the last save". An entry still matching its parse-time record is recorded nowhere, so the bookkeeping is the size of the edit. An entry with no baseline at all — appended after the load, or in a lexicon built from scratch — is stamped only where its dateModified is blank, so an exporter carrying real dates in from another data model keeps them. Its first save records a baseline either way, so a later edit to it does bump. iter_problems() does not stamp. It stays read-only, and its docstring says the bytes it validates precede the stamping a save does rather than claiming to be what save() would write; nothing generated is ever a finding, so validating first and saving after is sound. canonicalize() generates nothing either: sorting and reformatting change no entry's content. An unparseable date is residue rather than a date, so a stamp replaces it and the original string is dropped — a consequence pinned by a test and documented alongside the rest in docs/en/fidelity.md, with the guides that teach editing and building an export, and the streaming writer's note that it stamps nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five corrections to entry stamping. A document the byte scanner declined had no stamping baseline at all, so every undated entry read as new and a save that changed nothing dated all of them. Byte reuse needs the source bytes but stamping needs only the digests, so the reader now records digests and dates for such a document too, and a no-op save of one leaves it alone. when= is normalized to UTC at whole seconds, so an explicit moment reaches the output in the one form the rest of it uses instead of carrying an offset or fractional seconds through _fmt_date. A naive value is refused: read as UTC and read as local time it names moments hours apart, and picking one silently writes a date the caller did not mean. Stamping now commits with the write. stamp_entries returns what undoes it, save() and save_zip() put the dates and the baseline back when the write does not go through, and the pass decides before it mutates so that the one step that can fail — digesting content XML cannot represent — refuses with nothing stamped rather than half-stamped at whatever entry the refusal came from. The baseline dict is rebuilt each pass rather than updated in place, so an entry appended and later removed is no longer held alive, with its whole subtree, by a record nothing will consult again. Entries the document was loaded with are retained by their parse-time records as before. Entries are iterated by identity, so an entry aliased into the list twice is decided and stamped once — it is one object with one pair of dates, whatever its output happens to say twice. default_now() documents what seconds precision costs: one second holds one date, so a second edit saved inside the same second as the first carries the same stamp. Sub-second precision would buy the distinction at the cost of the form every consumer expects, and when= forces a distinct moment. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Internal vocabulary is out of the guides: a reader has no use for the word baseline, and the streaming note now says what the limitation is — a pass that never saw the document cannot tell which entries changed — rather than naming the machinery it lacks. Refused-write behavior is stated where it is a contract, in fidelity.md and on save(), and no longer repeated in a guide whose script aborts before it saves. The comments on the clock, the undo, the identity keying and the two-phase pass keep their reasons and drop the reassurance around them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two paths left a date the caller set unstampable for good. A save that stamps nothing still writes what the model holds, so a date the caller put there is the date now on disk. note_caller_dates records it, and the next stamping save measures a further edit against it. Without that the entry kept failing the stale test — its date differs from the load, which is exactly what a deliberate date looks like — and was never stamped again, while the same sequence through a stamping save bumped normally. An entry written unstamped under the date it was loaded with is deliberately not noted: its content is on disk under a date that no longer describes it, and the next stamping save should still say so. Dates are compared as the document will carry them rather than as moments. Two aware values an hour and an offset apart are the same instant and equal to ==, so restating 2008-12-12T09:42:48+10:00 at -05:00 — an edit, and an edit to the date itself — read as content changed without its date, and the stamp overwrote the one thing the caller had touched. The parse-time record lookup and the identity keying move into helpers shared by both passes, and _apply_stamps now always returns an undo. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
imnasnainaec
force-pushed
the
stamp-date-modified-on-save
branch
from
September 14, 2026 12:48
fadf3b3 to
908c35a
Compare
Lexicon.save() wrote the .lift and then rendered each companion in turn, so content XML cannot represent — a lone surrogate in a range label, say — was refused only once the .lift was already on disk, rewritten and stamped for a save that raised. The method's own contract says nothing is written and nothing is stamped in that case. The .lift, its companions, and the paths they go to are now all settled before the first byte lands, which is how save_zip has always worked: it stages the whole package in a temp directory and writes the archive once. Companion targets move into _companion_targets(), since the write loop now runs on rendered bytes and has nothing left to decide. Failure past the .lift write is unchanged: the file on disk carries those stamps, so they stand. This buys refusal, not atomicity. An OS error partway through the companions still leaves a mix of new and old files on disk; that would want staged writes and renames across every path that writes a document. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
note_caller_dates started from a copy of the whole previous baseline and only ever added to it, so a record for an entry appended and later removed survived every stamp=False save. _EntryRecord holds its Entry strongly, to keep that object's id() from being reused while the record stands, which means such a record also kept the removed entry's whole subtree alive until the next stamping save rebuilt the dict from the entries present. Filtering the copy to those same entries gives both passes one rule. Dropping the record is also what erases the only evidence that the date an appended entry carries was generated here rather than set deliberately, so an entry that leaves, is saved past, and comes back keeps the date it left with even after a further edit. That is the standing cost of holding nothing the lexicon no longer holds, and a test pins it so it reads as a decision rather than an oversight. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #10, taking Route 1:
save()stamps the entries it rewrites, by default.What changed
Lexicon.save()andLexicon.save_zip()take two keyword-only parameters:stamp: bool = True— setdateModifiedon every entry whose content moved while its date stayed where its baseline had it, and fill a blankdateCreatedwith the same moment.stamp=Falsewrites the model exactly as it stands.when: datetime | None = None— the moment to write, in place of the wall clock. Timezone-aware, normalized to UTC whole seconds, which is what makes stamped output byte-reproducible.Entries only, as the survey in #10 concluded. An entry's digest spans its whole subtree, so an edit to a gloss on a nested subsense stamps the entry containing it. The policy lives in one private method, so the other eight date-bearing types inherit it if they ever need it.
Three cases are left as they stand: an entry the caller dated deliberately, an entry created since the load that already carries a date, and an untouched entry — reordering included.
Decisions #10 left open
Entries with no baseline — appended after the load, or in a from-scratch lexicon — are stamped only where
dateModifiedis blank, rather than unconditionally as Route 1 words it.Unconditional stamping would overwrite dates an exporter carried in from its own data model, and would rewrite every date on every save of a from-scratch lexicon. Such an entry's first save records a baseline either way, so a later edit to a date-carrying one does bump.
iter_problems()stays read-only. Its docstring no longer claims to validate "whatsave()would write": it says the bytes it reports on precede the stamping a save does, and that nothing generated is ever a finding, so validate-then-save is sound.save_zip()stamps by default too. A package is the hand-off to the tools that reconcile ondateModified, so leaving it unstamped would reproduce the bug in the case that matters most.Baseline bookkeeping
The stamping baseline is deliberately not the parse-time one.
Lexicon._stamps, and the next save measures against that. Without it, a second round of edits on one loaded lexicon would ship unstamped: its date differs from the load, which is exactly what a caller-set date looks like.changed_entries()stop reporting a stamped entry after a save, contradicting its documented "always against the load, never against the most recentsave()" guarantee._EntryRecordgains thedateModifiedit held when the record was taken. That is what separates "the content changed and the date did not" from "the caller set the date deliberately".One fidelity consequence worth a look
An unparseable date (
dateModified="whenever") is residue, not a date, so the model field isNone. An edited entry therefore gets stamped, replacing the original string — which astamp=Falsesave would have preserved.The alternative, treating unparseable-as-present and skipping the stamp, would leave the worst data the only data that never gets a usable date. Pinned by a test and documented in
docs/en/fidelity.md.Write ordering
save()renders the.liftand every companion before writing any of them. Content XML cannot represent is then refused with nothing on disk and nothing stamped, wherever in the set it sits.save_zip()already staged its whole package this way.Cost
One extra canonical serialization pass over the entries per stamping save; Route 1's table claimed none.
Sharing digests with
render_documentwould mean threading a cache through it and through_validate/_zip, which seemed worse than documenting the pass the waychanged_entries()andchanges()already document theirs.Not in scope
SOURCE_DATE_EPOCHas the default clock source (#9).default_now()is the single place the clock is read, so that stays a one-function change on top.when=already makes the byte-exact tests here writable without monkeypatching.Tests and docs
tests/test_stamp.pyis new: the rule at depth, the three left-alone cases, the baseline across successive saves, byte-reproducibility underwhen=, bothsave_zippaths, and the residue case above. The corpus byte-identity and Hypothesis round-trip suites pass unchanged, which is the point — stamping is driven by content, so an unedited document is stamped nowhere.Docs:
docs/en/fidelity.md, the editing and build-an-export guides,validate.md,lift-export-interop.md,large-files.md(the streaming writer stamps nothing, and why),read-edit-write.md,index.md,README.md, and the[0.1.0]changelog section.🤖 Generated with Claude Code
This change is