fix(api): concept_versions enumerates the complete record set#22
Merged
Conversation
Zenodo's deposit search returns unstable, overlapping pages: a single paginated pass can hand back the same record twice AND miss others. Callers that dedup or take the latest version (archive/backfill dedup, apply-metadata, latest_version) were therefore acting on a partial slice — silently. In practice this made a whole-concept `apply-metadata --license-only` cover only ~490 of 492 vcell records per run, needing several rounds to converge. Now concept_versions unions records by id across repeated full passes until the set is complete: the target count comes from the public records API (which does report a total), with a converged (nothing-new) pass as the fallback when that total is unavailable, bounded by a pass cap. Verified live against the 492-record vcell concept: one call now returns 492 unique records, 0 duplicates (was duplicating/missing before). - api: union-until-complete concept_versions (dedup by id) - tests: overlapping-page union, target stop, draft exclusion, sort order Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BZx7w9BexQc9wWU5JoHYaF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Zenodo's deposit search returns unstable, overlapping pages — a single paginated pass can return the same record twice and miss others.
concept_versionsdid one pass and appended, so every caller that dedups or takes the latest version (archive/backfilldedup,apply-metadata,latest_version) could silently act on a partial slice.This bit us for real: a whole-concept
apply-metadata --license-onlyon vcell reported "490 updated, 0 failed" but left ~51 records untouched, and re-runs kept surfacing different subsets (51 → 15 → 7) because each enumeration was a different incomplete sample. The stragglers were only found by unioning reads until the count hit the known total (492).Fix
concept_versionsnow unions records by id across repeated full passes until the set is complete:total), so we know when we've seen every record — no unknown-unknowns;Dedup is by record id, so duplicates from overlapping pages collapse.
Verification
Live against the 492-record vcell concept: one call now returns 492 unique records, 0 duplicates (previously duplicated and missed records). This means
apply-metadataand archive dedup now cover the full concept in a single pass.ruff+mypy --strictclean; 43 unit tests pass (added overlapping-page union, target-stop, draft exclusion, and sort-order cases with stubbed pagination).Follow-up from the vcell license normalization, where this incompleteness was diagnosed.
🤖 Generated with Claude Code
https://claude.ai/code/session_01BZx7w9BexQc9wWU5JoHYaF