Isolate perf-sensitive missions onto their own nodes#406
Open
sisuresh wants to merge 2 commits into
Open
Conversation
Performance missions (Max TPS, Min Block Time) measure throughput / block time and are corrupted when pods from other supercluster runs share their nodes. As a step toward running multiple supercluster runs in parallel (stellar/pod-fsr#55), give these missions exclusive use of their nodes. Each run is already identified by a unique network nonce. Tag every pod with that nonce (label "run-nonce") and give isolated missions a required pod anti-affinity that repels any stellar-core pod whose run-nonce differs from this run's. Kubernetes evaluates required anti-affinity symmetrically, so a single term gives both directions of isolation: * this run's pods will not schedule onto a node already hosting another run's pods, and * no other run's pods will schedule onto a node hosting this run's pods, for as long as this run's pods are alive -- even if the other run did not opt into isolation. Isolation is controlled by a new MissionContext field, dedicatedNodes, which the four perf missions set on themselves. No CLI flag or Jenkinsfile change is needed. This does not remove the Jenkins locks; that is a separate follow-up.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds Kubernetes scheduling isolation for performance-sensitive supercluster missions by tagging pods with a per-run nonce label and (optionally) applying a required pod anti-affinity to keep different runs’ stellar-core pods from co-locating on the same node.
Changes:
- Introduces
MissionContext.dedicatedNodesto opt missions into node exclusivity behavior. - Adds a per-run
run-noncepod label and wires required pod anti-affinity (by hostname) for dedicated missions. - Enables
dedicatedNodes = truefor Max TPS / Min Block Time missions and adds unit tests asserting the emitted pod spec.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/FSLibrary/StellarMissionContext.fs | Adds dedicatedNodes flag to mission context. |
| src/FSLibrary/StellarKubeSpecs.fs | Adds per-run pod labels and optional required pod anti-affinity for dedicated missions; refactors node-affinity helper. |
| src/FSLibrary/StellarCoreCfg.fs | Defines the run-nonce label key constant. |
| src/FSLibrary/MissionMinBlockTimeMixed.fs | Enables dedicatedNodes for perf mission. |
| src/FSLibrary/MissionMinBlockTimeClassic.fs | Enables dedicatedNodes for perf mission. |
| src/FSLibrary/MissionMaxTPSMixed.fs | Enables dedicatedNodes for perf mission. |
| src/FSLibrary/MissionMaxTPSClassic.fs | Enables dedicatedNodes for perf mission. |
| src/FSLibrary.Tests/Tests.fs | Adds unit tests for nonce label + dedicated anti-affinity behavior. |
| src/App/Program.fs | Sets default dedicatedNodes = false in the main mission context. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+67
to
+71
| // When set, this run requires exclusive use of its nodes: its pods will | ||
| // not be scheduled onto a node hosting another run's pods, and no other | ||
| // run's pods will be scheduled onto its nodes while it is alive. Used for | ||
| // performance-sensitive missions (Max TPS, Min Block Time) whose | ||
| // measurements would be corrupted by co-tenant workloads. |
Comment on lines
+403
to
+405
| // A required pod anti-affinity that repels every supercluster pod belonging to | ||
| // a _different_ run, i.e. carrying a run-nonce label other than this run's. | ||
| // Kubernetes evaluates required anti-affinity symmetrically, so a single term |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Give perf-sensitive missions (
MaxTPSClassic,MaxTPSMixed,MinBlockTimeClassic,MinBlockTimeMixed) exclusive use of their nodes so pods from other runs can't skew their measurements. A step toward parallel runs (stellar/pod-fsr#55); does not touch the Jenkins locks.Each pod is now labeled with its run nonce (
run-nonce), and isolated missions get a required pod anti-affinity repelling anystellar-corepod with a different nonce. Kubernetes evaluates this symmetrically, so it both keeps this run off nodes with other runs' pods and keeps other runs off this run's nodes. Controlled by a newMissionContext.dedicatedNodesflag the four missions set themselves — no CLI/Jenkinsfile change.Tested via new unit tests and a live run on ssc-eks (pods got the label + anti-affinity and landed on distinct dedicated nodes).