Optimize MakeCurrentSnapshotsQuery: single grouped max() pass (SQLite) - #118
Draft
hahn-kev-bot wants to merge 1 commit into
Draft
hahn-kev-bot wants to merge 1 commit into
hahn-kev-bot wants to merge 1 commit into
Conversation
The current-snapshots query selected the latest snapshot per entity with a first_value() window function and then removed the duplicate rows it produced with GROUP BY s.EntityId. The plan sorted the snapshot/commit join twice: a temp B-tree for the window ORDER BY, then a second temp B-tree for the GROUP BY. Replace it with a single grouped aggregation. Scanning Snapshots via IX_Snapshots_EntityId delivers rows already grouped by EntityId, so GROUP BY needs no sort, and max() is a streaming aggregate, so there is no window sort either. SQLite guarantees that with exactly one max() aggregate the bare columns come from the row that produced the max, so "s".* is the snapshot whose commit is greatest. The commit order (DateTime, Counter, Id) is encoded as one lexically-ordered text key (Counter zero-padded to 20 digits, covering the full non-negative long range) so a single max() reproduces the tuple ordering. EXPLAIN drops both temp B-trees; the plan is now just a scan of Snapshots by EntityId plus the commit lookup. Measured on SQLite (50k entities): ~497ms vs ~786ms with history, ~324ms vs ~502ms after pruning (~35-37% faster). This relies on SQLite's bare-column behaviour and is not valid on other providers. Result set verified identical to the previous query for the no-filter case and for ignoreChangesAfter filters (including DateTime ties broken by Counter/Id) spanning early, middle and late history. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2nfncTWgKAjER7Ff4PUMS
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MakeCurrentSnapshotsQueryselects the newest snapshot per entity. It used afirst_value()window function and then removed the duplicate rows it produced withGROUP BY s.EntityId. The plan sorted the snapshot/commit join twice:This PR replaces it with a single grouped
max()aggregation.How it works
Two facts make the new query sort-free:
SnapshotsviaIX_Snapshots_EntityIddelivers rows already grouped byEntityId, soGROUP BYneeds no sort.max()is a streaming aggregate, so there is no window sort either.SQLite guarantees that when a query has exactly one
min()/max()aggregate, every bare (non-aggregated) column takes its value from the row that produced that max (docs). Sos.*is the whole snapshot row whose commit is greatest.max()needs a single scalar, so the commit order(DateTime, Counter, Id)is encoded as one lexically-ordered text key:DateTimeis stored by EF as fixed-width sortable text (the same assumption the oldORDER BYrelied on),Counteris a non-negativelongzero-padded to 20 digits (covers the fulllongrange) so text order matches numeric order, andId(a fixed-width GUID) is last.EXPLAIN QUERY PLANafter the change — both temp B-trees are gone:Provider note
This relies on SQLite's bare-column behaviour and is not valid on other providers (e.g. Postgres). The query is SQLite-only by design; a portable equivalent would be
ROW_NUMBER() ... WHERE rn = 1.Benchmarks
Measured with a standalone Python/SQLite harness (schema and indexes mirroring the EF model), 50k entities, best of several runs:
Correctness
The returned result set was verified identical to the previous query for:
ignoreChangesAfterfilters (includingDateTimeties broken byCounter/Id) spanning early, middle and late history.Testing
Validated at the SQL layer only — the .NET SDK was not available in the environment used to write this, so
dotnet test(theRepositoryTestscoverCurrentSnapshots) has not been run yet. Please run the suite before merging.Possible follow-up
Denormalizing the commit sort keys onto
Snapshotswith a composite index removes the commit join entirely (~3x in the same benchmark), but needs a schema migration — out of scope for this PR.🤖 Generated with Claude Code
https://claude.ai/code/session_01C2nfncTWgKAjER7Ff4PUMS
Generated by Claude Code