feat: report rows in DEFAULT that no maintenance run would reach

CHANGELOG.md

@@ -12,6 +12,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 * a backfilled window always attaches with writers held at the parent ([#89](https://github.com/bedrock-python/pg-partsmith/issues/89)) ([b1260c1](https://github.com/bedrock-python/pg-partsmith/commit/b1260c1a8bcdd6388c3836d57eb548f8fe5959bb))
 
+## 1.8.0 — rows in DEFAULT that no run would reach are reported
+
+An application that writes its own key values can put a row into DEFAULT after the table
+was adopted: an event dated last year lands there because no partition covers it. Creation
+walks forward from the cursor, so nothing ever came back for it. `plan` said there was
+nothing to do, `plan --check` exited 0, and retention could not drop what was never in a
+partition. Found by Outpost's maintainer: its publish API accepts a client-supplied `time`.
+
+`plan` now reports such rows as `rows_in_default`, a WARNING, so `plan --check` exits 3 and
+`apply` carries it in `result.issues`; `backfill` moves them into a partition of their own.
+What decides is the earliest row in DEFAULT: one in a window the same plan creates is moved
+by its attach, and one dated ahead of the cursor by the run that gets there, so neither is
+reported. During an adoption the history waiting for `backfill` is reported the same way,
+until it has moved. `ActualTree.default_earliest` carries the earliest key, read with the
+same index probe `partition_data` uses.
+
 ## 1.7.2 — a backfill's last attach no longer rejects a live insert
 
 The attach that ends a backfilled window is documented to hold writers at the parent, and

docs/guide/cli.md

@@ -347,6 +347,12 @@ rows per batch a month's rows were out of sight for up to 0.42 s and inserts wai
 128 ms; with batches larger than a month, out of sight for about 30 ms, and inserts waited
 up to 247 ms.
 
+Until `backfill` has moved the history, `plan` reports it as `rows_in_default` and `apply`
+exits `3` with the same finding. Once it has, the finding means rows arrived in DEFAULT
+later: an application that writes its own key values can date a row before every partition.
+If yours does, schedule `backfill` ahead of `apply`; with DEFAULT empty it does nothing and
+exits `0`.
+
 ## Commands around the lifecycle
 
 A document can name a command, or a block of Python, to run before a drop, after a

docs/guide/partition-existing-table.md

@@ -79,6 +79,8 @@ Each call:
 6. with DEFAULT drained, attaches any window's partition a previous run filled and never
    attached — the process died in between — which `plan` reports as `unattached_rows`.
 
+Until DEFAULT is drained, `plan` reports what is left in it as `rows_in_default`.
+
 `result.complete` says whether DEFAULT is drained; `result.partitions` lists what was
 created; `result.issues` explains anything that could not be handled. A call that runs out
 of budget mid-window leaves that partition detached and filled so far; the next call

docs/guide/troubleshooting.md

@@ -113,6 +113,21 @@ backfill back to the window. Run `backfill`: once DEFAULT is drained it looks fo
 these tables and attaches them. Only a table whose name is the one the scheme gives that
 window under this root counts — its rows are never read otherwise.
 
+## `rows_in_default` — rows no maintenance run will reach
+
+```text
+[warning] rows_in_default: public.events_default holds rows of 2025_06, a window behind the cursor that no partition of public.events covers, so no maintenance run will reach them, retention included; backfill (partition_data) moves them into partitions of their own.
+```
+
+Creation walks forward from the cursor, so a row in DEFAULT from a window before it stays
+there: no `apply` creates that window again, and retention, which drops partitions, never
+sees it. Two ways to get there: an adoption whose history `backfill` has not moved yet, and
+an application that writes its own key values — an event dated last year, say — after the
+table was adopted. Run `backfill`; it creates the window's partition and moves the rows,
+and retention then treats that partition like any other. The finding names the earliest such
+window. Rows of a window the same plan creates are left to its attach, and rows dated ahead
+of the cursor to the run that reaches them, so neither is reported.
+
 ## `grace_pending`, `drop_deferred` (INFO)
 
 A detached orphan waiting out its grace period, or one whose `DropAfter(when=…)`

docs/reference/findings.md

@@ -31,6 +31,7 @@ issues too, with the exception's name in front of the message.
 | `grace_pending` | INFO | A detached orphan still within its grace period. | Nothing. |
 | `drop_deferred` | INFO | An orphan past its grace whose drop condition does not hold yet. | Nothing. |
 | `unattached_rows` | WARNING | A table under a window's partition name holds rows but was never attached — a fill that stopped before its attach — so no query through the root sees them. | Run `backfill` (`partition_data`); it attaches the table. |
+| `rows_in_default` | WARNING | The root's DEFAULT partition holds rows of a window behind the cursor that has no partition: history not yet moved, or a row written with a key no partition covers. No maintenance run reaches them, retention included. | Run `backfill` (`partition_data`). |
 
 ## Operation reasons
 

pg_partsmith/aio/services/inspection.py

@@ -3,7 +3,7 @@
 from __future__ import annotations
 
 from datetime import UTC, datetime
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Any
 
 from pg_partsmith.boundaries import Axis, CursorSource, Window
 from pg_partsmith.planner import PlanMode, PlanningContext, fact_targets, unattached_window
@@ -44,6 +44,9 @@ async def inspect(self, config: TablePartitionConfig, *, measure: bool = True) -
         unattached = await unattached_tables(self._metadata, config)
         if unattached:
             tree = tree.model_copy(update={"unattached": unattached})
+        earliest = await default_earliest(self._metadata, config, tree)
+        if earliest is not None:
+            tree = tree.model_copy(update={"default_earliest": earliest})
 
         policy = config.lifecycle
         if not policy.needs_facts:
@@ -100,6 +103,22 @@ async def context(
         )
 
 
+async def default_earliest(metadata: PartitionMetadataProvider, config: TablePartitionConfig, tree: ActualTree) -> Any:
+    """The smallest leading-key value in the root's DEFAULT partition among rows whose whole key is set.
+
+    A row with a NULL in the key is routed to DEFAULT on purpose and stays
+    there; any other row belongs to a window that has no partition. One
+    index probe when the key is indexed.
+    """
+    root = config.scheme
+    if not isinstance(root, RangePartitioning):
+        return None
+    default = next((child for child in tree.root.children if child.is_default), None)
+    if default is None:
+        return None
+    return await metadata.get_leading_key_minimum(default.name, root.key)
+
+
 async def unattached_tables(
     metadata: PartitionMetadataProvider, config: TablePartitionConfig
 ) -> tuple[UnattachedTable, ...]:

pg_partsmith/plan.py

@@ -163,6 +163,10 @@ class FindingReason(StrEnum):
         UNATTACHED_ROWS: A partition named for a window of the root holds rows
             but was never attached -- a fill stopped before its attach -- so no
             query through the root sees them. ``partition_data`` attaches it.
+        ROWS_IN_DEFAULT: The root's DEFAULT partition holds rows of windows that
+            have no partition: history not yet moved, or rows written with a key
+            outside every partition. Retention never reaches them;
+            ``partition_data`` moves them into partitions of their own.
     """
 
     LEGACY_LEAF = "legacy_leaf"
@@ -186,6 +190,7 @@ class FindingReason(StrEnum):
     GRACE_PENDING = "grace_pending"
     DROP_DEFERRED = "drop_deferred"
     UNATTACHED_ROWS = "unattached_rows"
+    ROWS_IN_DEFAULT = "rows_in_default"
 
 
 # Reasons that describe a healthy, deliberate state (policy evolution, a

pg_partsmith/planner.py

@@ -246,6 +246,7 @@ def _owns(self, relkind: RelationKind) -> bool:
     def run(self) -> None:
         self._plan_level(self.config.scheme, self.actual.root, depth=0)
         self._report_unattached()
+        self._report_rows_in_default()
 
     def _report_unattached(self) -> None:
         """A partition that was filled and never attached holds rows no query through the root sees.
@@ -269,6 +270,37 @@ def _report_unattached(self) -> None:
                 f"through {self.config.qualified_name} sees them; backfill (partition_data) attaches it.",
             )
 
+    def _report_rows_in_default(self) -> None:
+        """Rows in the root's DEFAULT partition from a window behind the cursor that has no partition.
+
+        History an adoption has not moved yet, or a row written with a key
+        that no partition covers. Creation walks forward from the cursor, so
+        nothing would ever come back for them, retention included. Rows of a
+        window this plan creates are moved by its attach, and rows at or past
+        the cursor by a later one, so the earliest row is the one that decides.
+        """
+        earliest = self.actual.default_earliest
+        root = self.config.scheme
+        default = next((child for child in self.actual.root.children if child.is_default), None)
+        if earliest is None or default is None or not isinstance(root, RangePartitioning):
+            return
+        boundaries = root.range_boundaries
+        position = boundaries.decode(str(earliest))
+        window = boundaries.window_at(earliest if position is None else position)
+        cursor = self.ctx.now if boundaries.axis is Axis.TIME else self.ctx.cursors.get(root.key[0])
+        if cursor is not None and not window < boundaries.window_at(cursor):
+            return
+        planned = {op.target.rpartition(".")[2] for op in self.creates}
+        if boundaries.child_name(self.config.table_name, window) in planned:
+            return
+        self._record(
+            default.name,
+            FindingReason.ROWS_IN_DEFAULT,
+            f"{default.name} holds rows of {boundaries.describe(window)}, a window behind the cursor that no "
+            f"partition of {self.config.qualified_name} covers, so no maintenance run will reach them, retention "
+            "included; backfill (partition_data) moves them into partitions of their own.",
+        )
+
     # ── Dispatch ────────────────────────────────────────────────────────────────
 
     def _plan_level(self, level: SchemeBase, node: PartitionNode, *, depth: int) -> None:

pg_partsmith/sync/services/inspection.py

@@ -3,7 +3,7 @@
 from __future__ import annotations
 
 from datetime import UTC, datetime
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Any
 
 from pg_partsmith.boundaries import Axis, CursorSource, Window
 from pg_partsmith.planner import PlanMode, PlanningContext, fact_targets, unattached_window
@@ -44,6 +44,9 @@ def inspect(self, config: TablePartitionConfig, *, measure: bool = True) -> Actu
         unattached = unattached_tables(self._metadata, config)
         if unattached:
             tree = tree.model_copy(update={"unattached": unattached})
+        earliest = default_earliest(self._metadata, config, tree)
+        if earliest is not None:
+            tree = tree.model_copy(update={"default_earliest": earliest})
 
         policy = config.lifecycle
         if not policy.needs_facts:
@@ -100,6 +103,22 @@ def context(
         )
 
 
+def default_earliest(metadata: PartitionMetadataProvider, config: TablePartitionConfig, tree: ActualTree) -> Any:
+    """The smallest leading-key value in the root's DEFAULT partition among rows whose whole key is set.
+
+    A row with a NULL in the key is routed to DEFAULT on purpose and stays
+    there; any other row belongs to a window that has no partition. One
+    index probe when the key is indexed.
+    """
+    root = config.scheme
+    if not isinstance(root, RangePartitioning):
+        return None
+    default = next((child for child in tree.root.children if child.is_default), None)
+    if default is None:
+        return None
+    return metadata.get_leading_key_minimum(default.name, root.key)
+
+
 def unattached_tables(metadata: PartitionMetadataProvider, config: TablePartitionConfig) -> tuple[UnattachedTable, ...]:
     """The tables named under the root and attached to nothing, ``holds_rows`` answered where it matters.
 

pg_partsmith/topology.py

@@ -23,7 +23,7 @@
 from collections.abc import Collection, Sequence
 from datetime import datetime
 from enum import StrEnum
-from typing import Annotated, Literal
+from typing import Annotated, Any, Literal
 
 from pydantic import BaseModel, ConfigDict, Field, model_validator
 
@@ -418,13 +418,18 @@ class ActualTree(BaseModel):
         orphans: Marker-tagged detached tables whose marker names the root.
         unattached: Tables named under the root, attached to nothing and
             unmarked: partitions whose creation stopped before their attach.
+        default_earliest: The smallest leading-key value among the rows of the
+            root's DEFAULT partition whose whole partition key is set: where
+            its rows of windows with no partition begin. None when it holds
+            none, or when nobody looked.
     """
 
     model_config = ConfigDict(frozen=True)
 
     root: PartitionNode
     orphans: tuple[DetachedPartition, ...] = ()
     unattached: tuple[UnattachedTable, ...] = ()
+    default_earliest: Any = None
 
     def find(self, name: str) -> PartitionNode | None:
         """Return the attached node with schema-qualified ``name``, or None."""

tests/integration/aio/test_planning.py

@@ -113,6 +113,34 @@ async def test__apply__a_reader_holds_the_table__the_attach_gives_up_within_the_
     assert await scalar(db_engine, f'SELECT count(*) FROM "{table}_default"') == 0  # noqa: S608
 
 
+# ── rows in DEFAULT ─────────────────────────────────────────────────────────────
+
+
+async def test__plan__a_row_dated_before_every_partition__is_reported_until_backfill_moves_it(
+    db_engine: AsyncEngine, table: str
+) -> None:
+    # Arrange -- adopted, and then an event dated over a year back lands in DEFAULT
+    await exec_sql(db_engine, f'CREATE TABLE "{table}_default" PARTITION OF "{table}" DEFAULT')
+    config = monthly_config(table, create_ahead=1)
+    service = make_service(db_engine)
+    now = datetime.fromisoformat(NOW).replace(tzinfo=UTC)
+    await service.apply(config, await service.plan(config, now=now))
+    await exec_sql(db_engine, f"INSERT INTO \"{table}\" (created_at, payload) VALUES ('2025-06-15', 'late')")  # noqa: S608
+
+    # Act
+    before = await service.plan(config, now=now)
+    moved = await service.partition_data(config)
+    after = await service.plan(config, now=now)
+
+    # Assert
+    assert [(f.partition_name, f.reason, f.severity) for f in before.findings] == [
+        (f"public.{table}_default", FindingReason.ROWS_IN_DEFAULT, Severity.WARNING)
+    ]
+    assert moved.rows_moved == 1
+    assert after.findings == ()
+    assert await is_attached(db_engine, f"{table}__2025_06")
+
+
 # ── plan() / apply() ────────────────────────────────────────────────────────────
 
 

tests/integration/sync/test_planning.py

@@ -110,6 +110,34 @@ def test__apply__a_reader_holds_the_table__the_attach_gives_up_within_the_lock_t
     assert scalar(sync_db_engine, f'SELECT count(*) FROM "{table}_default"') == 0  # noqa: S608
 
 
+# ── rows in DEFAULT ─────────────────────────────────────────────────────────────
+
+
+def test__plan__a_row_dated_before_every_partition__is_reported_until_backfill_moves_it(
+    sync_db_engine: Engine, table: str
+) -> None:
+    # Arrange -- adopted, and then an event dated over a year back lands in DEFAULT
+    exec_sql(sync_db_engine, f'CREATE TABLE "{table}_default" PARTITION OF "{table}" DEFAULT')
+    config = monthly_config(table, create_ahead=1)
+    service = make_service(sync_db_engine)
+    now = datetime.fromisoformat(NOW).replace(tzinfo=UTC)
+    service.apply(config, service.plan(config, now=now))
+    exec_sql(sync_db_engine, f"INSERT INTO \"{table}\" (created_at, payload) VALUES ('2025-06-15', 'late')")  # noqa: S608
+
+    # Act
+    before = service.plan(config, now=now)
+    moved = service.partition_data(config)
+    after = service.plan(config, now=now)
+
+    # Assert
+    assert [(f.partition_name, f.reason, f.severity) for f in before.findings] == [
+        (f"public.{table}_default", FindingReason.ROWS_IN_DEFAULT, Severity.WARNING)
+    ]
+    assert moved.rows_moved == 1
+    assert after.findings == ()
+    assert is_attached(sync_db_engine, f"{table}__2025_06")
+
+
 # ── plan() / apply() ────────────────────────────────────────────────────────────
 
 

tests/unit/sync/test_inspection.py

@@ -22,7 +22,7 @@
 )
 from pg_partsmith.planner import PlanMode
 from pg_partsmith.scheme import HashPartitioning, RangePartitioning
-from pg_partsmith.sync.services.inspection import PartitionInspector, unattached_tables
+from pg_partsmith.sync.services.inspection import PartitionInspector, default_earliest, unattached_tables
 from pg_partsmith.topology import ActualTree, DetachedPartition, FactKind, UnattachedTable
 
 NOW = datetime(2024, 3, 15, 12, 0, tzinfo=UTC)
@@ -304,6 +304,53 @@ def test__unattached_tables__a_root_that_is_not_range__is_not_searched(metadata:
     metadata.get_unattached_tables.assert_not_called()
 
 
+# ── rows in DEFAULT ─────────────────────────────────────────────────────────────
+
+
+def _default() -> PartitionNode:
+    return PartitionNode(name="events_default", parent_name="events", level=1, bounds={"kind": "default"})  # type: ignore[arg-type]
+
+
+def test__inspect__default_partition_with_rows__its_earliest_key_is_on_the_tree(
+    inspector: PartitionInspector, metadata: MagicMock
+) -> None:
+    # Arrange
+    metadata.get_actual_tree.return_value = _tree(_default())
+    metadata.get_leading_key_minimum.return_value = datetime(2023, 6, 15, tzinfo=UTC)
+
+    # Act
+    tree = inspector.inspect(_config())
+
+    # Assert
+    assert tree is not None
+    assert tree.default_earliest == datetime(2023, 6, 15, tzinfo=UTC)
+    metadata.get_leading_key_minimum.assert_called_once_with("events_default", ("created_at",))
+
+
+def test__default_earliest__no_default_partition__nothing_is_read(metadata: MagicMock) -> None:
+    # Arrange
+    tree = _tree(_child("events__2024_02", "2024-02-01", "2024-03-01"))
+
+    # Act
+    earliest = default_earliest(metadata, _config(), tree)
+
+    # Assert
+    assert earliest is None
+    metadata.get_leading_key_minimum.assert_not_called()
+
+
+def test__default_earliest__a_root_that_is_not_range__is_not_read(metadata: MagicMock) -> None:
+    # Arrange
+    config = TablePartitionConfig(table_name="tasks", scheme=HashPartitioning(key="task_id", modulus=4))
+
+    # Act
+    earliest = default_earliest(metadata, config, _tree(_default()))
+
+    # Assert
+    assert earliest is None
+    metadata.get_leading_key_minimum.assert_not_called()
+
+
 # ── context ─────────────────────────────────────────────────────────────────────