xds-client/retry: add jitter to connection backoff#2738
Open
W4lspirit wants to merge 2 commits into
Open
Conversation
Randomize each backoff delay by a configurable factor (gRFC A6) so reconnecting clients don't retry in lockstep. Default jitter is 0.2 (±20%); `with_jitter` validates the factor is in [0.0, 1.0] and 0.0 disables it. Adds the `rand` dependency.
YutaoMa
reviewed
Jul 20, 2026
| self.attempt += 1; | ||
| Some(duration) | ||
| let factor = 1.0 + self.policy.jitter * rand::rng().random_range(-1.0..1.0); | ||
| Some(base.mul_f64(factor)) |
Contributor
There was a problem hiding this comment.
negative value would cause panic here, and because currently RetryPolicy fields are public, the [0.0, 1.0] range isn't a type level invariant. grpc-go clamps backoff to be non-negative, but I'm good with privatize RetryPolicy and enforce invariant there if you want.
Author
Contributor
|
Thanks for the PR! All logic and tests looks great to me, my only comment is regarding the negative delay issue above. Feel free to address it and ping me again for approval. |
Even after constructing a valid retry policy, a user could still manually mutate any field with invalid values.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The reconnect backoff was fully deterministic: every client that lost its xDS stream at the same time (control-plane restart or a network blip) computed the identical delay schedule and retried in lockstep. That synchronized retry storm hammers the management server right when it's least able to absorb it.
Solution
Randomize each delay by a configurable factor per gRFC A6 so clients spread their reconnects across a window instead of colliding. Default jitter is 0.2 (±20%) ;
with_jittervalidates the factor is in [0.0, 1.0], and 0.0 restores the old deterministic behavior.Random
I hesitated between
randcrate andfastrand, in the end I fallback to the same dep grpc is already using for the jitter in named_resolution