Skip to content

Batch recursive ZK deletes and recover partial instance drops - #427

Open
PranaviAncha wants to merge 7 commits into
linkedin:devfrom
PranaviAncha:praa/batched-delete-recursively
Open

PranaviAncha wants to merge 7 commits into
linkedin:devfrom
PranaviAncha:praa/batched-delete-recursively

Conversation

@PranaviAncha

@PranaviAncha PranaviAncha commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Issues

  • My PR addresses the following Helix issues and references them in the PR description:

Based on #173, which this PR supersedes, with the review feedback on it addressed.

Description

  • Here are some details about my PR, including screenshots of any UI changes:

Note for REST clients: DELETE /clusters/{cluster}/instances/{instance} on an instance that does not exist now returns 404. On dev it returns 400 (the missing getInstanceConfig threw into badRequest). A client that treats 400 as "already gone" must treat 404 that way instead; 400 now means the drop failed and should be retried. 409 means the instance was re-added while it was being dropped; the re-added instance is kept, and the drop must not be retried.

Summary: Delete large instance subtrees in batches that stay under jute.maxbuffer, keep the drop atomic for every instance small enough to fit in one batch, clean up after the rare partial drop of a large instance, and never delete an instance re-added while it is being dropped.

Why: dropInstance deleted the InstanceConfig and the whole /INSTANCES/{instance} subtree in one multi(). On instances with about 13K MESSAGES the packet was about 3.75 MB; ZK rejected it, the client saw a connection loss, and the default 24h ZkClient retry pinned 1,880 Helix REST threads.

1. dropInstance: atomic when it fits, two steps when it doesn't

ZKHelixAdmin.dropInstance runs two steps, each retried on its own.

  • Step 1 deletes the InstanceConfig. It lists the instance subtree first. If config plus subtree fit in one batch (DELETE_RECURSIVE_BATCH_SIZE, 1,000 ops, covering almost every instance), they are deleted in one atomic multi(), as on dev, so a failed drop leaves nothing behind. Otherwise only the config is deleted, so the controller immediately stops counting the instance (auto maintenance mode budget, delayed rebalance, capacity).
  • Step 2 deletes only the leftover subtree, fenced (see 3), reusing step 1's listing on its first attempt. It never deletes a config: once step 1 is done, any config it sees belongs to a re-added instance, which it keeps, and the drop throws HelixConflictException.
  • The atomic or batched choice is made before sending: ZK reports an oversized multi() as a connection loss, which ZkClient resends until its operation timeout instead of failing.
  • NotEmpty, NoNode and a lost fence (while there is no config) are retried up to 3 times, listing the subtree again each time; other errors fail fast. The precondition fails only when both the config and the subtree are gone, so calling it again finishes a partial drop.

2. Batched ZkClient.deleteRecursively

The subtree is walked children first, then deleted in multi() batches of at most 1,000 ops (about 240 KB). Every caller benefits, including dropCluster and ZkBaseDataAccessor.deleteRecursively.

  • Deleting a missing path is still a no-op, and the call is still not atomic.
  • Missing znodes are skipped, as with the old per-node delete: if a batch fails with NoNode, for example because a znode was deleted concurrently, or the batch committed but its response was lost and the resend found the znodes gone, that batch is redone one znode at a time and missing znodes are skipped.
  • A child created under a znode being deleted still fails with ZkClientException -> ZkException -> KeeperException.NotEmptyException, the shape callers retry on.
  • A failure while walking the subtree (e.g. an unreadable child list) is wrapped in ZkClientException like a failed delete, so dropInstance reports it as a HelixException (REST 400) instead of a 500.

3. Fencing against a concurrent re-add

Every leftover subtree delete (step 2 of a drop, addInstance, the garbage collector) first writes empty data to /INSTANCES/{instance}, which only bumps its data version, then checks that the instance has no config, then deletes the subtree in batches that each start with an Op.check on that version.

  • If the instance is re-added before the fence, the config check finds it and nothing is deleted.
  • If it is re-added after the fence, the re-add recreates /INSTANCES/{instance}, so its version no longer matches. The remaining batches fail their check and abort without deleting any of the new znodes.
  • Either way the re-added instance is kept and the caller gets HelixConflictException (REST 409).

4. Recovering from a partial drop of a large instance

A large drop that fails between its steps leaves the subtree without a config.

Caller Behavior with only the subtree left
Controller InstanceGarbageCollector (new) The leader runs it on its own thread, independent of cluster events and pipelines, every ClusterConfig.INSTANCE_GARBAGE_COLLECTION_INTERVAL_MS (new, 1 minute by default; 0 or negative turns it off for the cluster). It finds such subtrees from ZK and deletes them once seen for a minute, so partial drops by any caller are cleaned up, including after a controller failover. Uses the fenced HelixAdmin.deleteDroppedInstanceGarbage (new), so a re-added or live instance is never touched. A failure on one instance does not stop the others.
addInstance, REST PUT /clusters/{c}/instances/{i}, auto-join participants Deletes the leftover subtree (fenced), then adds the instance with a clean subtree. If the instance is live, throws HelixConflictException (REST 409).
dropInstance, REST DELETE Finishes the drop (200), or keeps an instance re-added meanwhile (HelixConflictException, REST 409).
ClusterSetup.dropInstanceFromCluster (CLI, legacy webapp, swapInstance) Finishes the drop instead of reporting the instance as missing, through deleteDroppedInstanceGarbage, so an instance re-added meanwhile is kept.
getInstancesInCluster, getInstancesInClusterWithTag Omit the instance, so e.g. rebalance cannot assign partitions to it.

Tests

  • The following tests are written for this issue:

zookeeper-api TestRawZkClient:

  • testDeleteRecursivelyLargeSubtree: 2,500 children deleted in multi() calls of 1000, 1000 and 501 ops.
  • testDeleteRecursivelyDeepSubtree: 1,204 znodes over 3 levels, deleted in calls of 1000 and 204 ops, so a batch boundary falls between levels.
  • testDeleteRecursivelyResumesAfterBatchFailure: the second batch fails, the first stays deleted, and a second call deletes the rest.
  • testDeleteRecursivelyToleratesConcurrentDelete: a znode deleted between the walk and its batch is skipped.
  • testDeleteRecursivelyToleratesResentBatch: a batch resent after it committed does not fail the call.
  • testDeleteRecursivelyNotEmptyKeepsRetryableExceptionShape: a child created concurrently surfaces as the retryable NotEmpty shape.
  • testDeleteRecursivelyWrapsListingFailure: a failure while walking the subtree is wrapped in ZkClientException.
  • testDeleteInBatchesStopsWhenFenceChanges, testDeleteInBatchesReturnsWhenFenceIsGone: a fenced delete aborts once the fence znode is recreated, and returns once it is gone.
  • testDeleteRecursivelyOnMissingPathIsNoOp, testDeleteRecursivelySkipsChildDeletedBeforeWalk.

helix-core TestZkHelixAdmin:

  • testDropInstanceIsAtomicUnlessSubtreeExceedsOneBatch: a small instance is deleted in one multi() that includes the config; a large one deletes the config, then fences the subtree root, then deletes the subtree in batches of at most 1,000 ops that each check the fence first.
  • testDropInstanceKeepsInstanceReAddedDuringDrop: an instance re-added between batches keeps its config and new subtree, and the drop reports a conflict.
  • testDropInstanceRetryKeepsInstanceReAddedAfterConfigDelete: a large drop loses its fence to another deleter, the instance is re-added before the retry, and the retry keeps it and reports a conflict instead of dropping it again.
  • testDropInstanceWithLargeMessageSubtree, testDropInstanceWithDeepSubtreeShape: end to end.
  • testDropInstanceResumesAfterPartialDelete: config gone, subtree left, drop completes.
  • testDropInstanceRetriesOnNoNode, testDropInstanceGivesUpAfterRepeatedNoNode, testDropInstanceCompletesWhenZnodeDeletedMidDrop: NoNode on the atomic delete is retried, up to 3 times.
  • testDropInstanceFailsFastOnNonRetryableError, testDropInstanceReportsSubtreeListingFailure: other errors are not retried and surface as HelixException.
  • testAddInstanceCleansLeftoverSubtree, testAddInstanceRejectsLeftoverSubtreeOfLiveInstance, testAddInstanceSucceedsAfterDropClearsLeftoverSubtree.
  • testInstanceListsExcludeLeftoverSubtree: instance lists omit a leftover instance.
  • testZkHelixAdmin: the NotEmpty retry mock now targets deleteRecursivelyAtomic, with the same 3 retry semantics as dev.

helix-core TestInstanceGarbageCollector (real ZK):

  • testDeletesLeftoverSubtreeAfterGracePeriod: a leftover is deleted only after a full grace period; configured and live instances are kept.
  • testKeepsInstanceReAddedOrLiveSinceFirstSeen: an instance re-added or come live after it was first seen is not deleted.
  • testOnlyLeaderCollects: a non-leader deletes nothing and forgets what it has seen.
  • testLeaderControllerSchedulesCollection: a started leader controller schedules the collection without any cluster event.
  • testIntervalFromClusterConfig: 0 or negative turns the collection off; turned back on, a leftover still waits a full grace period and runs follow the configured interval.
  • testFailureOnOneLeftoverDoesNotStopOthers: an unexpected error deleting one leftover does not stop the others from being collected.

helix-core TestClusterSetup:

  • testDropInstanceFromClusterResumesPartialDrop.
  • testDropInstanceFromClusterKeepsInstanceReAddedDuringPartialDrop: an instance re-added before ClusterSetup finishes a partial drop is kept, with a conflict.

helix-rest TestPerInstanceAccessor:

  • testDeleteInstanceResumesPartialDrop: DELETE finishes a partial drop (200), then returns 404; PUT cleans a leftover and adds the instance (200); PUT on a live instance's leftover returns 409.
  • testDeleteInstanceReturnsConflictWhenReAddedDuringDrop: DELETE returns 409 when the drop finds the instance re-added.

The batching, NoNode, atomic-choice, fencing, re-add, 409, ClusterSetup and addInstance tests fail against the previous code and pass with this change.

  • The following is the result of the "mvn test" command on the appropriate module:

Run with JDK 11 against this branch, after mvn -pl helix-rest -am clean install -DskipTests:

mvn -pl zookeeper-api test -Dtest=TestRawZkClient
[INFO] Tests run: 34, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

mvn -pl helix-core test -Dtest='TestZkHelixAdmin,TestClusterSetup,TestInstanceGarbageCollector,TestParticipantDeregistrationStage,TestClusterRebalanceTimer,TestClusterConfig'
[INFO] Tests run: 152, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

mvn -pl helix-rest test -Dtest=TestPerInstanceAccessor
[INFO] Tests run: 55, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

Changes that Break Backward Compatibility (Optional)

  • My PR contains changes that break backward compatibility or previous assumptions for certain methods or API. They include:

  • REST DELETE /clusters/{cluster}/instances/{instance} returns 404 instead of 400 when the instance does not exist, 409 when the instance is re-added while it is being dropped, and succeeds when only the subtree remains.

  • dropInstance of an instance whose config and subtree exceed one batch is no longer atomic. A failure between the steps leaves the subtree without a config until a caller or the controller's garbage collector finishes the drop.

  • dropInstance fails with "does not exist in config" only when both the config and the subtree are gone; the separate "does not exist in instances" error is removed.

  • addInstance deletes a leftover subtree without an InstanceConfig before adding, instead of creating the config on top of it; it throws HelixConflictException (REST 409 instead of 500) if that instance is live.

  • getInstancesInCluster no longer lists instances without an InstanceConfig, and getInstancesInClusterWithTag skips them instead of throwing IllegalStateException.

  • dropInstance throws HelixConflictException (a HelixException) when the instance is re-added while it is being dropped, and keeps the re-added instance.

  • ClusterSetup.dropInstanceFromCluster finishes a partial drop instead of throwing "does not exist".

  • The leader controller runs a new periodic task, InstanceGarbageCollector, that lists /INSTANCES once a minute by default. New ClusterConfig field INSTANCE_GARBAGE_COLLECTION_INTERVAL_MS sets the interval; 0 or negative turns it off.

  • New public API: HelixAdmin.deleteDroppedInstanceGarbage (default method, implemented by ZKHelixAdmin), and getSubtreeChildrenFirst / deleteInBatches on RealmAwareZkClient and its implementations.

Documentation (Optional)

  • In case of new functionality, my PR adds documentation in the following wiki page:

N/A. The deleteRecursively javadoc documents the batching, atomicity, and exception shapes.

Commits

  • My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Code Quality

  • My diff has been formatted using helix-style.xml
    (helix-style-intellij.xml if IntelliJ IDE is used)

@PranaviAncha PranaviAncha changed the title Batch ZkClient.deleteRecursively to stay under jute.maxbuffer Batch recursive ZK deletes and recover partial instance drops Nov 11, 2026
Comment on lines +325 to +330
// Phase 1
if (_zkClient.exists(instanceConfigPath)) {
_zkClient.delete(instanceConfigPath);
}
// Phase 2
_zkClient.deleteRecursively(instancePath);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry I missed this in the old PR. how about keeping the atomic delete for the common case like 99.9% and only going two-phase when the subtree is too big? the walk already gives us the op count, so if config + subtree fits in one DELETE_RECURSIVE_BATCH_SIZE batch we send a single multi like today and never leave a partial drop

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 71ba341. Config plus subtree within one batch are deleted in a single atomic multi(); only larger instances go two-phase.

// Phase 2
_zkClient.deleteRecursively(instancePath);
return;
} catch (ZkClientException e) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh wait, zk has no separate exception for this. a multi over jute.maxbuffer makes the server close the socket, so the client just sees connectio loss. and then ZkClient.multi goes through retryUntilConnected and keeps resending it for up to 24h, so this catch only sees it a day ;( later as a ZkTimeoutException. so I think the atomic vs batched choice has to happen before sending based on the op count

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The choice is made from the op count before sending, so an oversized multi() is never sent.

static final long PARTIAL_DROP_RETRY_DELAY_MS = 60 * 1000L;

// Instances whose drop by this stage failed, retried by resumePartialDrops.
private final Set<String> _partialDropsToResume = ConcurrentHashMap.newKeySet();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is essentially noOp unless participant de-registration is enabled right, and ig we had discussed caveats of enabling that for stateful pods.
My original proposal for this type of deletion was to have a separate async stage and not club it with this.

also _partialDropsToResume is in memory only, so a controller failover forgets every pending resume, and REST, ClusterSetup and purge drops never get into it. I think finding leftovers from ZK (INSTANCES children with no CONFIGS/PARTICIPANT entry) would cover all of them.

@PranaviAncha PranaviAncha Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted. The leader controller now runs InstanceGarbageCollector every minute on its own thread, outside the pipelines. It finds leftovers from ZK, so it covers every caller and survives failover.

// config means a drop is partial or still in flight; re-adding would inherit its stale state.
String instancePath = PropertyPathBuilder.instance(clusterName, nodeId);
if (_zkClient.exists(instancePath)) {
throw new HelixConflictException("Node " + nodeId + " has a residual subtree at "

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a thought, how about having addInstance finish the leftover cleanup when no live instance exists, instead of throwing? with auto-join, ParticipantManager.joinCluster sees isInstanceSetup false (no config), calls addInstance and now gets the conflict, so the participant can't connect until someone redoes the drop by hand. on dev it just created over the old subtree.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. addInstance deletes the leftover subtree (fenced) unless the instance is live (then 409).

}
// multi() throws on any failed op, so a successful return means the whole batch was deleted.
try {
multi(ops);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deleteRecursively should treat NoNode as already deleted, say by redoing that batch one node at a time and skipping the gone ones. the old per-node delete ignored it. now if a batch commits but its reply is lost to a connection loss, retryUntilConnected resends it, hits NoNode and the whole call fails. that's 12 call sites in 8 files, incl. ParticipantManager.removePreviousTaskCurrentStates on every new session.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. A batch that hits NoNode is redone one znode at a time, skipping missing ones.

private void dropInstanceZnodesOnce(String clusterName, String instanceName) {
String instanceConfigPath = PropertyPathBuilder.instanceConfig(clusterName, instanceName);
String instancePath = PropertyPathBuilder.instance(clusterName, instanceName);
List<String> subtree = _zkClient.getSubtreeChildrenFirst(instancePath);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we pick full drop vs garbage-only once per drop? I think after phase 1 deletes the config, any config the drop sees later can only be a re-add, so retries should only use deleteDroppedInstanceGarbageOnce. right now each retry checks the config again, so a re-add gets wiped even with no client DELETE, like a deregistration drop of a host with 2000 messages while it restarts and auto-joins

1. drop:         deletes config, fences root (v1)
2. addInstance:  no config, so it starts cleaning the leftover subtree, fences root (v2)
3. drop batch:   check(v1) fails, no config yet, so it retries
4. drop retry:   lists the subtree
5. addInstance:  done, writes new config + new subtree
6. drop retry:   sees a config, does a full drop, new instance is gone, drop returns fine

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 34591eb. dropInstance now deletes the config first, then only the leftover subtree, so a retry can't do a full drop and a re-added instance is kept with a conflict. ClusterSetup had the same gap, fixed in 52b0ba5.

@PranaviAncha
PranaviAncha force-pushed the praa/batched-delete-recursively branch from b7ac914 to f31d310 Compare November 16, 2026 15:54
PranaviAncha and others added 2 commits November 16, 2026 21:25
dropInstance deleted the InstanceConfig and the whole instance subtree
in one multi(). On instances with ~13K MESSAGES the packet exceeded
jute.maxbuffer, ZK reported CONNECTIONLOSS, and the 24h ZkClient retry
pinned Helix REST threads. The non-atomic deleteRecursively avoided the
limit but issued one round trip per znode.

deleteRecursively now walks the subtree children first and deletes it
in multi() batches of 1000 ops. A child created concurrently keeps the
NotEmpty exception shape callers retry on, and a failure while walking
the subtree is wrapped in ZkClientException like a failed batch.

dropInstance deletes the InstanceConfig first, then the subtree, and
retries NotEmpty and NoNode up to 3 times. NoNode also covers a batch
that committed but whose response was lost and was resent.

A drop that fails partway leaves a subtree without a config, so:
- REST DELETE finishes such a drop, and returns 404 when neither the
  config nor the subtree exists.
- addInstance throws HelixConflictException on a leftover subtree, and
  REST PUT maps it to 409.
- ClusterSetup.dropInstanceFromCluster finishes the drop instead of
  refusing.
- ParticipantDeregistrationStage retries its own failed drops.
- getInstancesInCluster lists only instances with a config, and
  getInstancesInClusterWithTag skips instead of throwing.

Based on linkedin#173.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
dropInstance deletes the InstanceConfig and the /INSTANCES/{instance} subtree in
one atomic multi() when they fit in one batch. A larger instance deletes the
config together with a fence on the subtree root, then the subtree in batches
that each check the fence first, so an instance re-added during the drop keeps
its new znodes and the drop reports a HelixConflictException. The subtree is
listed once per attempt.

ZkClient.deleteRecursively redoes a batch one node at a time on NoNode, and
exposes getSubtreeChildrenFirst and deleteInBatches with an optional fence.

addInstance deletes a leftover subtree without an InstanceConfig before adding,
unless the instance is live (HelixConflictException).

Add InstanceGarbageCollector, which the leader controller runs every minute on
its own thread, independent of cluster events and pipelines. It deletes
leftover subtrees seen for at least a minute through the new fenced
HelixAdmin.deleteDroppedInstanceGarbage. getLeaderController skips a
controller that has not handled its first event yet instead of throwing a
NullPointerException.

Correct the jute.maxbuffer default in the batch size javadoc.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@PranaviAncha
PranaviAncha force-pushed the praa/batched-delete-recursively branch from f31d310 to 903a8c9 Compare November 16, 2026 15:56
*/
class InstanceGarbageCollector {
private static final Logger LOG = LoggerFactory.getLogger(InstanceGarbageCollector.class);
static final long COLLECTION_INTERVAL_MS = 60 * 1000L;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we make the interval a ClusterConfig field, default 1 min, with 0 or negative turning it off? I think 1 min is fine since a sweep is just 3 getChildren, but today there is no way to turn it off for a cluster, or to stop the first sweep after upgrade from deleting every old leftover across all clusters at once

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in fd9ba7e. Added INSTANCE_GARBAGE_COLLECTION_INTERVAL_MS, 1 min by default, with 0 or negative turning it off. It's read from ZK on every run.

dropInstance now runs in two steps, each retried on its own. Step 1
deletes the InstanceConfig, together with the subtree when they fit in
one multi(). Step 2 deletes only the leftover subtree. A retry can no
longer go back to a full drop after the config is gone, so a config
seen in step 2 belongs to a re-added instance: it is kept and the drop
reports HelixConflictException instead of dropping it again.

Step 2 now takes the fence on the subtree root, so step 1 no longer
writes it together with the config delete. Its first attempt reuses
step 1's subtree listing.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@PranaviAncha
PranaviAncha force-pushed the praa/batched-delete-recursively branch 2 times, most recently from b7240de to c96a8a7 Compare November 19, 2026 14:19
Add INSTANCE_GARBAGE_COLLECTION_INTERVAL_MS to ClusterConfig, 1 minute by
default, with 0 or negative turning the collection off for the cluster.
This lets a cluster opt out, and lets operators turn it off before an
upgrade and back on cluster by cluster, instead of every cluster deleting
its old leftovers in the first sweep.

Each run reads the setting from ZK and schedules the next run after that
interval, so a change applies from the next run without depending on the
controller's cache. While it is off, a run only rereads the setting, once
a minute.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@PranaviAncha
PranaviAncha force-pushed the praa/batched-delete-recursively branch from c96a8a7 to fd9ba7e Compare November 19, 2026 14:23
InstanceConfig instanceConfig = admin.getInstanceConfig(clusterId, instanceName);
admin.dropInstance(clusterId, instanceConfig);
admin.dropInstance(clusterId, new InstanceConfig(instanceName));
} catch (HelixException e) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we map HelixConflictException to 409 here like PUT does? with the retry fix its now the normal result when the instance is re-added mid drop, but it still lands in badRequest, and the description says 400 means the drop failed and should be retried. so a client that retries can drop the instance that was just re-added, e.g. one added back by PUT before it goes live

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 3d1d0c5. DELETE now maps HelixConflictException to 409, like PUT.

PranaviAncha and others added 3 commits November 21, 2026 10:27
dropInstance throws HelixConflictException when the instance is re-added
while it is being dropped, and keeps the re-added instance. DELETE mapped
it to 400, which tells callers the drop failed and should be retried, so
a retry could drop the instance that was just re-added. Map it to 409, as
PUT already does.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
ClusterSetup.dropInstanceFromCluster finishes a drop that left a subtree
without an InstanceConfig. It called dropInstance, so an instance re-added
after ClusterSetup read the config was dropped while enabled, bypassing
its rule that an instance must be disabled before it is dropped. Call
deleteDroppedInstanceGarbage instead, which keeps a re-added instance and
throws HelixConflictException.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Keep scheduling the collection after an Error, which ended it until
  leadership changed.
- Contain any failure deleting one leftover to that instance, so the
  others are still collected in the same run.
- Name the operation in retry logs and errors, so cleaning a leftover
  subtree no longer reports "Failed to drop instance".
- Rename deleteInstanceConfigOnce to deleteConfigAndSmallSubtreeOnce,
  isFenceIntact to checkFenceUnchanged, fenceOp and fenceVersion to
  newFenceOp and getFenceVersion, and isInstanceGarbageCollectionRunning
  to isInstanceGarbageCollectionScheduled.
- Document the new behavior of HelixAdmin.getInstancesInCluster and
  dropInstance, and that DELETE_RECURSIVE_BATCH_SIZE counts the fence check.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants