[Excelius-Wang]

Summary

• Allow exact-dedup document priorities in the range [0, 65535].
• Replace unsigned negation with priority_max - priority so that zero remains lower priority than positive values.
• Raise ValueError for out-of-range priorities and add regression coverage for zero priorities, ties, and invalid boundaries.

Motivation

document_priority can legitimately return zero. For example, the URL deduplication example uses len(doc.text) // 4, which returns zero for documents shorter than four characters.

The current assertion rejects these documents. Simply removing it would also be incorrect because negating an unsigned zero leaves it at zero, causing it to sort before positive priorities.

Fixes #337

Testing

• python -m pytest -q tests/pipeline/dedup
  • 23 passed, 3 skipped, 2 subtests passed
• make quality

---

Note

Medium Risk
Changes which duplicate document is kept when priority is 0 or near range limits, but behavior is localized to exact-dedup signature ordering and is covered by new regression tests.

Overview
Exact dedup now accepts document_priority values in [0, 65535] instead of rejecting zero. Stage-1 signature sorting no longer negates unsigned priorities (which broke ordering when priority was 0); it uses priority_max - priority before and after sort so higher user priorities still win among duplicate hashes.

Validation in save_hashes raises ValueError for out-of-range priorities instead of an assert. Tests cover zero-priority URL dedup (keeps higher priority / one doc per URL cluster), tied zero priorities, and boundaries -1 / 65536.

^{Reviewed by Cursor Bugbot for commit e4f39b7. Bugbot is set up for automated code reviews on this repo. Configure here.}

[lpla]

I just checked this PR applied to current main. For a 100-document fixture with priorities 1, 2, 42 and 65535, the signature files were byte-identical before and after applying this PR, at both 32 and 64-bit hash precision and with 1/3 finder workers.

I also mixed a shard written by the old implementation with a shard written by this PR containing zero priorities. Both the old and new finder and filter implementations retained the expected highest-priority documents. The PR's ten exact-dedup tests also passed.

This supports compatibility for existing positive-priority signatures in those cases. These checks used xxhash 3.6.0 on Python 3.12 in my environment.

[Excelius-Wang]

Thanks @lpla for checking compatibility against current main, including mixed old/new shards!

Could a maintainer take a look when available? This fixes the zero-priority case reported in #337. Happy to address any review feedback.