Append commit instead of individual transactions to commitlog#4404
Open
Append commit instead of individual transactions to commitlog#4404
Conversation
This moves the following responsibilities to the datastore: - maintenance of the transaction offset - deciding how many transactions are in a commit
Allowing to restore `Committed` return
We don't really need batched transactions at the moment, so avoid the boxed array allocation. Durability::append_tx takes a `Transaction`, though, requiring the offset to be supplied by the datastore.
kim
added a commit
that referenced
this pull request
Feb 26, 2026
Instead of periodic flush + sync, the simplified commitlog buffering makes it easy to just pop a batch of transaction from the queue, commit them, and then flush + sync after each batch. This may lead to an overall higher number of fsyncs on the system, but also adapts to the tx throughput of the individual database (up to the batch size). After taking some measurements, we may want to make the batch size configurable at runtime. Depends-on: #4404
Contributor
|
Is this same patch as reverted one or anything else? |
jsdt
approved these changes
Feb 27, 2026
crates/durability/src/imp/local.rs
Outdated
|
|
||
| data = txdata_rx.recv() => { | ||
| let Some(txdata) = data else { | ||
| tx = transactions_rx.recv() => { |
Contributor
There was a problem hiding this comment.
Should we use a recv_many here (which will receive up to some number of messages, but always complete as soon as there are any messages)? It seems like that would give us the eager flushing behavior we want without actually flushing for every transaction.
Contributor
Author
There was a problem hiding this comment.
Yeah, sorry for the stacked PRs -- this is done via #4478
Instead of periodic flush + sync, the simplified commitlog buffering makes it easy to just pop a batch of transaction from the queue, commit them, and then flush + sync after each batch. This may lead to an overall higher number of fsyncs on the system, but also adapts to the tx throughput of the individual database (up to the batch size). After taking some measurements, we may want to make the batch size configurable at runtime. Depends-on: #4404
github-merge-queue bot
pushed a commit
that referenced
this pull request
Feb 28, 2026
# Description of Changes This does 3 things to the keynote-2 benchmark: - it changes the default alpha to 1.5, which we actually tested the other services with - it turns on confirmed reads (not the default for < 2.0) - it removes warmup This was tested on #4404 to have no impact on the TPS throughput of spacetimedb. **This PR shouldn't be merged before #4404 has.** # API and ABI breaking changes None # Expected complexity level and risk 1 # Testing This tweaks a bench test.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-open #4140 (reverted in #4292).
The original patch was merged a bit too eagerly.
It should go in after 2.0 is released with some confidence.