Repository navigation
device: serialize nonce stamping with the outbound enqueue - #103
Merged
Merged
Conversation
illotum
requested review from
cmol,
creachadair,
danderson and
jwhited
and
a lite review from Copilot
September 25, 2026 17:30
jwhited
approved these changes
Sep 25, 2026
illotum
force-pushed
the
illotum/nonce-multiqueue
branch
from
September 28, 2026 20:46
85ed634 to
5be9a3a
Compare
SendStagedPackets stamps keypair.sendNonce and enqueues to
peer.queue.outbound as two steps. Two TUN readers staging for the same
peer can invert them, and the sequential sender then puts those packets
on the wire out of nonce order.
Pinning the read side to match, which would keep stamping serial without
a lock, is not open to us: the kernel hashes the 5-tuple with a siphash
keyed by an unexported random value that userspace cannot reproduce, and
overriding it requires TUNSETSTEERINGEBPF.
The following was measured on builds instrumented to observe reorder
events and max reorder distance. 12-core hosts, TX, 25s runs, 3 rounds,
a custom load generator rotating at speed through UDP flows:
2 conn / 2 tun queues, 1 peer x 16 streams, ~1000 new flows/s
before 20.36 Gb/s 3.892 cores/Gb/s reordered 2107, 1645, 1448
after 20.58 Gb/s 3.885 cores/Gb/s reordered 0, 0, 0
Max reorder distance was 47.
Updates: tailscale/corp#37878
Signed-off-by: Alex Valiushko <alexvaliushko@tailscale.com>
Change-Id: I0894d39a928aadc28a58939e7125f9a26a6a6964
illotum
force-pushed
the
illotum/nonce-multiqueue
branch
from
September 28, 2026 21:28
5be9a3a to
e4c0118
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SendStagedPackets stamps keypair.sendNonce and enqueues to peer.queue.outbound as two steps. Two TUN readers staging for the same peer can invert them, and the sequential sender then puts those packets on the wire out of nonce order.
Pinning the read side to match, which would keep stamping serial without a lock, is not open to us: the kernel hashes the 5-tuple with a siphash keyed by an unexported random value that userspace cannot reproduce, and overriding it requires TUNSETSTEERINGEBPF.
The following was measured on builds instrumented to observe reorder events and max reorder distance. 12-core hosts, TX, 25s runs, 3 rounds, a custom load generator rotating at speed through UDP flows:
2 conn / 2 tun queues, 1 peer x 16 streams, ~1000 new flows/s
before 20.36 Gb/s 3.892 cores/Gb/s reordered 2107, 1645, 1448
after 20.58 Gb/s 3.885 cores/Gb/s reordered 0, 0, 0
Max reorder distance was 47.
Updates: tailscale/corp#37878
Change-Id: I0894d39a928aadc28a58939e7125f9a26a6a6964