Skip to content

Add opt-in best-effort placement mode to WAGED rebalancer - #442

Open
LZD-PratyushBhatt wants to merge 2 commits into
devfrom
lzd/waged-best-effort-placement
Open

LZD-PratyushBhatt wants to merge 2 commits into
devfrom
lzd/waged-best-effort-placement

Conversation

@LZD-PratyushBhatt

Copy link
Copy Markdown
Collaborator

Adds ClusterConfigProperty.WAGED_BEST_EFFORT_PLACEMENT_ENABLED (default false) so WAGED places as many replicas as hard constraints allow instead of failing the rebalance on CAPACITY_DEFICIT or NO_CANDIDATE_NODE in skewed fault-zone or zero-spare topologies.

When enabled:

  • Clamps negative scoring-capacity estimates, orders replicas by partition ordinal and capacity-feasible active home within each state-priority tier, and breaks equal candidate scores by fault-zone candidate count.
  • Runs bounded single-hop relocation when a replica has no direct candidate node, keeping enabled-live replicas off disabled or offline delayed targets.
  • Replaces relocated modeled instances cleanly in delayed minActiveReplica overwrite merges, preserves sole surviving copies on non-modeled instances when zero replicas fit, enforces cluster MAX_PARTITIONS_PER_INSTANCE on bootstrap admission, and keeps admin what-if and drain probes all-or-nothing.
  • Takes precedence over WAGED_INSTANCE_TAG_ISOLATION_ENABLED when both flags are set. With the flag off, WAGED and all guardrails behave as on dev.

Testing Done

TestWagedBestEffortPlacement covers skewed fault-zone cold traps, zero-spare removal, skewed scale-up, single-hop relocation cost and live-target rules, delayed-overwrite merges, cluster MAX_PARTITIONS_PER_INSTANCE swaps, EVACUATE sole-copy preservation, trimmer change detection, and 600-node tag-clique bounds.
Existing WAGED, tag-isolation, trimmer, and guardrail suites pass (160 tests), and helix-core + helix-rest compile with --release 8 and --release 11.

Add ClusterConfigProperty.WAGED_BEST_EFFORT_PLACEMENT_ENABLED (default
false) so WAGED places as many replicas as hard constraints allow
instead of aborting on CAPACITY_DEFICIT or NO_CANDIDATE_NODE in skewed
fault-zone or zero-spare topologies.

When enabled:
- Clamp negative scoring-capacity estimates, order replicas by
  partition ordinal and capacity-feasible active home within each
  state-priority tier, and break equal candidate scores by fault-zone
  candidate count.
- Run bounded single-hop relocation when a replica has no direct
  candidate node, keeping enabled-live replicas off disabled or
  offline delayed targets.
- Preserve sole surviving copies on non-modeled instances without
  retaining deleted or zero-replica partitions, update delayed
  overwrite merges for relocated live instances, and enforce cluster
  maxPartitionsPerInstance during bootstrap admission.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant