feat: report rows in DEFAULT that no maintenance run would reach (#94)

* feat: report rows in DEFAULT that no maintenance run would reach

A row written with a key outside every partition after adoption lands
in DEFAULT, and if its window is behind the cursor nothing ever comes
back for it: no apply creates that window again, retention cannot drop
it, and plan said there was nothing to do. Outpost's publish API takes
a client-supplied time, which is how its maintainer found it.

plan now reports it as rows_in_default (WARNING), so plan --check exits
3 and apply carries it in result.issues; backfill moves the rows into a
partition of their own. The earliest row in DEFAULT decides, read with
the index probe partition_data already uses: rows of a window the same
plan creates are left to its attach, and rows ahead of the cursor to
the run that reaches them. ActualTree.default_earliest carries it.

* test: a past window the plan names is left to its attach

---------

Co-authored-by: Alexey Shalaev <shalaevad.alexey@gmail.com>

Author: AlexeyShalaev

CHANGELOG.md

@@ -12,6 +12,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 * a backfilled window always attaches with writers held at the parent ([#89](https://github.com/bedrock-python/pg-partsmith/issues/89)) ([b1260c1](https://github.com/bedrock-python/pg-partsmith/commit/b1260c1a8bcdd6388c3836d57eb548f8fe5959bb))
 
+## 1.8.0 — rows in DEFAULT that no run would reach are reported
+
+An application that writes its own key values can put a row into DEFAULT after the table
+was adopted: an event dated last year lands there because no partition covers it. Creation
+walks forward from the cursor, so nothing ever came back for it. `plan` said there was
+nothing to do, `plan --check` exited 0, and retention could not drop what was never in a
+partition. Found by Outpost's maintainer: its publish API accepts a client-supplied `time`.
+
+`plan` now reports such rows as `rows_in_default`, a WARNING, so `plan --check` exits 3 and
+`apply` carries it in `result.issues`; `backfill` moves them into a partition of their own.
+What decides is the earliest row in DEFAULT: one in a window the same plan creates is moved
+by its attach, and one dated ahead of the cursor by the run that gets there, so neither is
+reported. During an adoption the history waiting for `backfill` is reported the same way,
+until it has moved. `ActualTree.default_earliest` carries the earliest key, read with the
+same index probe `partition_data` uses.
+
 ## 1.7.2 — a backfill's last attach no longer rejects a live insert
 
 The attach that ends a backfilled window is documented to hold writers at the parent, and

docs/guide/cli.md

@@ -347,6 +347,12 @@ rows per batch a month's rows were out of sight for up to 0.42 s and inserts wai
 128 ms; with batches larger than a month, out of sight for about 30 ms, and inserts waited
 up to 247 ms.
 
+Until `backfill` has moved the history, `plan` reports it as `rows_in_default` and `apply`
+exits `3` with the same finding. Once it has, the finding means rows arrived in DEFAULT
+later: an application that writes its own key values can date a row before every partition.
+If yours does, schedule `backfill` ahead of `apply`; with DEFAULT empty it does nothing and
+exits `0`.
+
 ## Commands around the lifecycle
 
 A document can name a command, or a block of Python, to run before a drop, after a

docs/guide/partition-existing-table.md

@@ -79,6 +79,8 @@ Each call:
 6. with DEFAULT drained, attaches any window's partition a previous run filled and never
    attached — the process died in between — which `plan` reports as `unattached_rows`.
 
+Until DEFAULT is drained, `plan` reports what is left in it as `rows_in_default`.
+
 `result.complete` says whether DEFAULT is drained; `result.partitions` lists what was
 created; `result.issues` explains anything that could not be handled. A call that runs out
 of budget mid-window leaves that partition detached and filled so far; the next call

docs/guide/troubleshooting.md

@@ -113,6 +113,21 @@ backfill back to the window. Run `backfill`: once DEFAULT is drained it looks fo
 these tables and attaches them. Only a table whose name is the one the scheme gives that
 window under this root counts — its rows are never read otherwise.
 
+## `rows_in_default` — rows no maintenance run will reach
+
+```text
+[warning] rows_in_default: public.events_default holds rows of 2025_06, a window behind the cursor that no partition of public.events covers, so no maintenance run will reach them, retention included; backfill (partition_data) moves them into partitions of their own.
+```
+
+Creation walks forward from the cursor, so a row in DEFAULT from a window before it stays
+there: no `apply` creates that window again, and retention, which drops partitions, never
+sees it. Two ways to get there: an adoption whose history `backfill` has not moved yet, and
+an application that writes its own key values — an event dated last year, say — after the
+table was adopted. Run `backfill`; it creates the window's partition and moves the rows,
+and retention then treats that partition like any other. The finding names the earliest such
+window. Rows of a window the same plan creates are left to its attach, and rows dated ahead
+of the cursor to the run that reaches them, so neither is reported.
+
 ## `grace_pending`, `drop_deferred` (INFO)
 
 A detached orphan waiting out its grace period, or one whose `DropAfter(when=…)`

docs/reference/findings.md

@@ -31,6 +31,7 @@ issues too, with the exception's name in front of the message.
 | `grace_pending` | INFO | A detached orphan still within its grace period. | Nothing. |
 | `drop_deferred` | INFO | An orphan past its grace whose drop condition does not hold yet. | Nothing. |
 | `unattached_rows` | WARNING | A table under a window's partition name holds rows but was never attached — a fill that stopped before its attach — so no query through the root sees them. | Run `backfill` (`partition_data`); it attaches the table. |
+| `rows_in_default` | WARNING | The root's DEFAULT partition holds rows of a window behind the cursor that has no partition: history not yet moved, or a row written with a key no partition covers. No maintenance run reaches them, retention included. | Run `backfill` (`partition_data`). |
 
 ## Operation reasons
 

pg_partsmith/aio/services/inspection.py

@@ -3,7 +3,7 @@
 from __future__ import annotations
 
 from datetime import UTC, datetime
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Any
 
 from pg_partsmith.boundaries import Axis, CursorSource, Window
 from pg_partsmith.planner import PlanMode, PlanningContext, fact_targets, unattached_window
@@ -44,6 +44,9 @@ async def inspect(self, config: TablePartitionConfig, *, measure: bool = True) -
         unattached = await unattached_tables(self._metadata, config)
         if unattached:
             tree = tree.model_copy(update={"unattached": unattached})
+        earliest = await default_earliest(self._metadata, config, tree)
+        if earliest is not None:
+            tree = tree.model_copy(update={"default_earliest": earliest})
 
         policy = config.lifecycle
         if not policy.needs_facts:
@@ -100,6 +103,22 @@ async def context(
         )
 
 
+async def default_earliest(metadata: PartitionMetadataProvider, config: TablePartitionConfig, tree: ActualTree) -> Any:
+    """The smallest leading-key value in the root's DEFAULT partition among rows whose whole key is set.
+
+    A row with a NULL in the key is routed to DEFAULT on purpose and stays
+    there; any other row belongs to a window that has no partition. One
+    index probe when the key is indexed.
+    """
+    root = config.scheme
+    if not isinstance(root, RangePartitioning):
+        return None
+    default = next((child for child in tree.root.children if child.is_default), None)
+    if default is None:
+        return None
+    return await metadata.get_leading_key_minimum(default.name, root.key)
+
+
 async def unattached_tables(
     metadata: PartitionMetadataProvider, config: TablePartitionConfig
 ) -> tuple[UnattachedTable, ...]:

pg_partsmith/plan.py

@@ -163,6 +163,10 @@ class FindingReason(StrEnum):
         UNATTACHED_ROWS: A partition named for a window of the root holds rows
             but was never attached -- a fill stopped before its attach -- so no
             query through the root sees them. ``partition_data`` attaches it.
+        ROWS_IN_DEFAULT: The root's DEFAULT partition holds rows of windows that
+            have no partition: history not yet moved, or rows written with a key
+            outside every partition. Retention never reaches them;
+            ``partition_data`` moves them into partitions of their own.
     """
 
     LEGACY_LEAF = "legacy_leaf"
@@ -186,6 +190,7 @@ class FindingReason(StrEnum):
     GRACE_PENDING = "grace_pending"
     DROP_DEFERRED = "drop_deferred"
     UNATTACHED_ROWS = "unattached_rows"
+    ROWS_IN_DEFAULT = "rows_in_default"
 
 
 # Reasons that describe a healthy, deliberate state (policy evolution, a

pg_partsmith/planner.py

@@ -246,6 +246,7 @@ def _owns(self, relkind: RelationKind) -> bool:
     def run(self) -> None:
         self._plan_level(self.config.scheme, self.actual.root, depth=0)
         self._report_unattached()
+        self._report_rows_in_default()
 
     def _report_unattached(self) -> None:
         """A partition that was filled and never attached holds rows no query through the root sees.
@@ -269,6 +270,37 @@ def _report_unattached(self) -> None:
                 f"through {self.config.qualified_name} sees them; backfill (partition_data) attaches it.",
             )
 
+    def _report_rows_in_default(self) -> None:
+        """Rows in the root's DEFAULT partition from a window behind the cursor that has no partition.
+
+        History an adoption has not moved yet, or a row written with a key
+        that no partition covers. Creation walks forward from the cursor, so
+        nothing would ever come back for them, retention included. Rows of a
+        window this plan creates are moved by its attach, and rows at or past
+        the cursor by a later one, so the earliest row is the one that decides.
+        """
+        earliest = self.actual.default_earliest
+        root = self.config.scheme
+        default = next((child for child in self.actual.root.children if child.is_default), None)
+        if earliest is None or default is None or not isinstance(root, RangePartitioning):
+            return
+        boundaries = root.range_boundaries
+        position = boundaries.decode(str(earliest))
+        window = boundaries.window_at(earliest if position is None else position)
+        cursor = self.ctx.now if boundaries.axis is Axis.TIME else self.ctx.cursors.get(root.key[0])
+        if cursor is not None and not window < boundaries.window_at(cursor):
+            return
+        planned = {op.target.rpartition(".")[2] for op in self.creates}
+        if boundaries.child_name(self.config.table_name, window) in planned:
+            return
+        self._record(
+            default.name,
+            FindingReason.ROWS_IN_DEFAULT,
+            f"{default.name} holds rows of {boundaries.describe(window)}, a window behind the cursor that no "
+            f"partition of {self.config.qualified_name} covers, so no maintenance run will reach them, retention "
+            "included; backfill (partition_data) moves them into partitions of their own.",
+        )
+
     # ── Dispatch ────────────────────────────────────────────────────────────────
 
     def _plan_level(self, level: SchemeBase, node: PartitionNode, *, depth: int) -> None:

pg_partsmith/sync/services/inspection.py

@@ -3,7 +3,7 @@
 from __future__ import annotations
 
 from datetime import UTC, datetime
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Any
 
 from pg_partsmith.boundaries import Axis, CursorSource, Window
 from pg_partsmith.planner import PlanMode, PlanningContext, fact_targets, unattached_window
@@ -44,6 +44,9 @@ def inspect(self, config: TablePartitionConfig, *, measure: bool = True) -> Actu
         unattached = unattached_tables(self._metadata, config)
         if unattached:
             tree = tree.model_copy(update={"unattached": unattached})
+        earliest = default_earliest(self._metadata, config, tree)
+        if earliest is not None:
+            tree = tree.model_copy(update={"default_earliest": earliest})
 
         policy = config.lifecycle
         if not policy.needs_facts:
@@ -100,6 +103,22 @@ def context(
         )
 
 
+def default_earliest(metadata: PartitionMetadataProvider, config: TablePartitionConfig, tree: ActualTree) -> Any:
+    """The smallest leading-key value in the root's DEFAULT partition among rows whose whole key is set.
+
+    A row with a NULL in the key is routed to DEFAULT on purpose and stays
+    there; any other row belongs to a window that has no partition. One
+    index probe when the key is indexed.
+    """
+    root = config.scheme
+    if not isinstance(root, RangePartitioning):
+        return None
+    default = next((child for child in tree.root.children if child.is_default), None)
+    if default is None:
+        return None
+    return metadata.get_leading_key_minimum(default.name, root.key)
+
+
 def unattached_tables(metadata: PartitionMetadataProvider, config: TablePartitionConfig) -> tuple[UnattachedTable, ...]:
     """The tables named under the root and attached to nothing, ``holds_rows`` answered where it matters.
 

pg_partsmith/topology.py

@@ -23,7 +23,7 @@
 from collections.abc import Collection, Sequence
 from datetime import datetime
 from enum import StrEnum
-from typing import Annotated, Literal
+from typing import Annotated, Any, Literal
 
 from pydantic import BaseModel, ConfigDict, Field, model_validator
 
@@ -418,13 +418,18 @@ class ActualTree(BaseModel):
         orphans: Marker-tagged detached tables whose marker names the root.
         unattached: Tables named under the root, attached to nothing and
             unmarked: partitions whose creation stopped before their attach.
+        default_earliest: The smallest leading-key value among the rows of the
+            root's DEFAULT partition whose whole partition key is set: where
+            its rows of windows with no partition begin. None when it holds
+            none, or when nobody looked.
     """
 
     model_config = ConfigDict(frozen=True)
 
     root: PartitionNode
     orphans: tuple[DetachedPartition, ...] = ()
     unattached: tuple[UnattachedTable, ...] = ()
+    default_earliest: Any = None
 
     def find(self, name: str) -> PartitionNode | None:
         """Return the attached node with schema-qualified ``name``, or None."""