Repository navigation
Batch recursive ZK deletes and recover partial instance drops - #427
PranaviAncha wants to merge 7 commits into
Conversation
| // Phase 1 | ||
| if (_zkClient.exists(instanceConfigPath)) { | ||
| _zkClient.delete(instanceConfigPath); | ||
| } | ||
| // Phase 2 | ||
| _zkClient.deleteRecursively(instancePath); |
There was a problem hiding this comment.
sorry I missed this in the old PR. how about keeping the atomic delete for the common case like 99.9% and only going two-phase when the subtree is too big? the walk already gives us the op count, so if config + subtree fits in one DELETE_RECURSIVE_BATCH_SIZE batch we send a single multi like today and never leave a partial drop
There was a problem hiding this comment.
Done in 71ba341. Config plus subtree within one batch are deleted in a single atomic multi(); only larger instances go two-phase.
| // Phase 2 | ||
| _zkClient.deleteRecursively(instancePath); | ||
| return; | ||
| } catch (ZkClientException e) { |
There was a problem hiding this comment.
oh wait, zk has no separate exception for this. a multi over jute.maxbuffer makes the server close the socket, so the client just sees connectio loss. and then ZkClient.multi goes through retryUntilConnected and keeps resending it for up to 24h, so this catch only sees it a day ;( later as a ZkTimeoutException. so I think the atomic vs batched choice has to happen before sending based on the op count
There was a problem hiding this comment.
Done. The choice is made from the op count before sending, so an oversized multi() is never sent.
| static final long PARTIAL_DROP_RETRY_DELAY_MS = 60 * 1000L; | ||
|
|
||
| // Instances whose drop by this stage failed, retried by resumePartialDrops. | ||
| private final Set<String> _partialDropsToResume = ConcurrentHashMap.newKeySet(); |
There was a problem hiding this comment.
this is essentially noOp unless participant de-registration is enabled right, and ig we had discussed caveats of enabling that for stateful pods.
My original proposal for this type of deletion was to have a separate async stage and not club it with this.
also _partialDropsToResume is in memory only, so a controller failover forgets every pending resume, and REST, ClusterSetup and purge drops never get into it. I think finding leftovers from ZK (INSTANCES children with no CONFIGS/PARTICIPANT entry) would cover all of them.
There was a problem hiding this comment.
Reverted. The leader controller now runs InstanceGarbageCollector every minute on its own thread, outside the pipelines. It finds leftovers from ZK, so it covers every caller and survives failover.
| // config means a drop is partial or still in flight; re-adding would inherit its stale state. | ||
| String instancePath = PropertyPathBuilder.instance(clusterName, nodeId); | ||
| if (_zkClient.exists(instancePath)) { | ||
| throw new HelixConflictException("Node " + nodeId + " has a residual subtree at " |
There was a problem hiding this comment.
a thought, how about having addInstance finish the leftover cleanup when no live instance exists, instead of throwing? with auto-join, ParticipantManager.joinCluster sees isInstanceSetup false (no config), calls addInstance and now gets the conflict, so the participant can't connect until someone redoes the drop by hand. on dev it just created over the old subtree.
There was a problem hiding this comment.
Done. addInstance deletes the leftover subtree (fenced) unless the instance is live (then 409).
| } | ||
| // multi() throws on any failed op, so a successful return means the whole batch was deleted. | ||
| try { | ||
| multi(ops); |
There was a problem hiding this comment.
deleteRecursively should treat NoNode as already deleted, say by redoing that batch one node at a time and skipping the gone ones. the old per-node delete ignored it. now if a batch commits but its reply is lost to a connection loss, retryUntilConnected resends it, hits NoNode and the whole call fails. that's 12 call sites in 8 files, incl. ParticipantManager.removePreviousTaskCurrentStates on every new session.
There was a problem hiding this comment.
Done. A batch that hits NoNode is redone one znode at a time, skipping missing ones.
| private void dropInstanceZnodesOnce(String clusterName, String instanceName) { | ||
| String instanceConfigPath = PropertyPathBuilder.instanceConfig(clusterName, instanceName); | ||
| String instancePath = PropertyPathBuilder.instance(clusterName, instanceName); | ||
| List<String> subtree = _zkClient.getSubtreeChildrenFirst(instancePath); |
There was a problem hiding this comment.
can we pick full drop vs garbage-only once per drop? I think after phase 1 deletes the config, any config the drop sees later can only be a re-add, so retries should only use deleteDroppedInstanceGarbageOnce. right now each retry checks the config again, so a re-add gets wiped even with no client DELETE, like a deregistration drop of a host with 2000 messages while it restarts and auto-joins
1. drop: deletes config, fences root (v1)
2. addInstance: no config, so it starts cleaning the leftover subtree, fences root (v2)
3. drop batch: check(v1) fails, no config yet, so it retries
4. drop retry: lists the subtree
5. addInstance: done, writes new config + new subtree
6. drop retry: sees a config, does a full drop, new instance is gone, drop returns fine
b7ac914 to
f31d310
Compare
dropInstance deleted the InstanceConfig and the whole instance subtree in one multi(). On instances with ~13K MESSAGES the packet exceeded jute.maxbuffer, ZK reported CONNECTIONLOSS, and the 24h ZkClient retry pinned Helix REST threads. The non-atomic deleteRecursively avoided the limit but issued one round trip per znode. deleteRecursively now walks the subtree children first and deletes it in multi() batches of 1000 ops. A child created concurrently keeps the NotEmpty exception shape callers retry on, and a failure while walking the subtree is wrapped in ZkClientException like a failed batch. dropInstance deletes the InstanceConfig first, then the subtree, and retries NotEmpty and NoNode up to 3 times. NoNode also covers a batch that committed but whose response was lost and was resent. A drop that fails partway leaves a subtree without a config, so: - REST DELETE finishes such a drop, and returns 404 when neither the config nor the subtree exists. - addInstance throws HelixConflictException on a leftover subtree, and REST PUT maps it to 409. - ClusterSetup.dropInstanceFromCluster finishes the drop instead of refusing. - ParticipantDeregistrationStage retries its own failed drops. - getInstancesInCluster lists only instances with a config, and getInstancesInClusterWithTag skips instead of throwing. Based on linkedin#173. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
dropInstance deletes the InstanceConfig and the /INSTANCES/{instance} subtree in
one atomic multi() when they fit in one batch. A larger instance deletes the
config together with a fence on the subtree root, then the subtree in batches
that each check the fence first, so an instance re-added during the drop keeps
its new znodes and the drop reports a HelixConflictException. The subtree is
listed once per attempt.
ZkClient.deleteRecursively redoes a batch one node at a time on NoNode, and
exposes getSubtreeChildrenFirst and deleteInBatches with an optional fence.
addInstance deletes a leftover subtree without an InstanceConfig before adding,
unless the instance is live (HelixConflictException).
Add InstanceGarbageCollector, which the leader controller runs every minute on
its own thread, independent of cluster events and pipelines. It deletes
leftover subtrees seen for at least a minute through the new fenced
HelixAdmin.deleteDroppedInstanceGarbage. getLeaderController skips a
controller that has not handled its first event yet instead of throwing a
NullPointerException.
Correct the jute.maxbuffer default in the batch size javadoc.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
f31d310 to
903a8c9
Compare
| */ | ||
| class InstanceGarbageCollector { | ||
| private static final Logger LOG = LoggerFactory.getLogger(InstanceGarbageCollector.class); | ||
| static final long COLLECTION_INTERVAL_MS = 60 * 1000L; |
There was a problem hiding this comment.
can we make the interval a ClusterConfig field, default 1 min, with 0 or negative turning it off? I think 1 min is fine since a sweep is just 3 getChildren, but today there is no way to turn it off for a cluster, or to stop the first sweep after upgrade from deleting every old leftover across all clusters at once
There was a problem hiding this comment.
Done in fd9ba7e. Added INSTANCE_GARBAGE_COLLECTION_INTERVAL_MS, 1 min by default, with 0 or negative turning it off. It's read from ZK on every run.
dropInstance now runs in two steps, each retried on its own. Step 1 deletes the InstanceConfig, together with the subtree when they fit in one multi(). Step 2 deletes only the leftover subtree. A retry can no longer go back to a full drop after the config is gone, so a config seen in step 2 belongs to a re-added instance: it is kept and the drop reports HelixConflictException instead of dropping it again. Step 2 now takes the fence on the subtree root, so step 1 no longer writes it together with the config delete. Its first attempt reuses step 1's subtree listing. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
b7240de to
c96a8a7
Compare
Add INSTANCE_GARBAGE_COLLECTION_INTERVAL_MS to ClusterConfig, 1 minute by default, with 0 or negative turning the collection off for the cluster. This lets a cluster opt out, and lets operators turn it off before an upgrade and back on cluster by cluster, instead of every cluster deleting its old leftovers in the first sweep. Each run reads the setting from ZK and schedules the next run after that interval, so a change applies from the next run without depending on the controller's cache. While it is off, a run only rereads the setting, once a minute. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
c96a8a7 to
fd9ba7e
Compare
| InstanceConfig instanceConfig = admin.getInstanceConfig(clusterId, instanceName); | ||
| admin.dropInstance(clusterId, instanceConfig); | ||
| admin.dropInstance(clusterId, new InstanceConfig(instanceName)); | ||
| } catch (HelixException e) { |
There was a problem hiding this comment.
can we map HelixConflictException to 409 here like PUT does? with the retry fix its now the normal result when the instance is re-added mid drop, but it still lands in badRequest, and the description says 400 means the drop failed and should be retried. so a client that retries can drop the instance that was just re-added, e.g. one added back by PUT before it goes live
There was a problem hiding this comment.
Done in 3d1d0c5. DELETE now maps HelixConflictException to 409, like PUT.
dropInstance throws HelixConflictException when the instance is re-added while it is being dropped, and keeps the re-added instance. DELETE mapped it to 400, which tells callers the drop failed and should be retried, so a retry could drop the instance that was just re-added. Map it to 409, as PUT already does. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
ClusterSetup.dropInstanceFromCluster finishes a drop that left a subtree without an InstanceConfig. It called dropInstance, so an instance re-added after ClusterSetup read the config was dropped while enabled, bypassing its rule that an instance must be disabled before it is dropped. Call deleteDroppedInstanceGarbage instead, which keeps a re-added instance and throws HelixConflictException. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Keep scheduling the collection after an Error, which ended it until leadership changed. - Contain any failure deleting one leftover to that instance, so the others are still collected in the same run. - Name the operation in retry logs and errors, so cleaning a leftover subtree no longer reports "Failed to drop instance". - Rename deleteInstanceConfigOnce to deleteConfigAndSmallSubtreeOnce, isFenceIntact to checkFenceUnchanged, fenceOp and fenceVersion to newFenceOp and getFenceVersion, and isInstanceGarbageCollectionRunning to isInstanceGarbageCollectionScheduled. - Document the new behavior of HelixAdmin.getInstancesInCluster and dropInstance, and that DELETE_RECURSIVE_BATCH_SIZE counts the fence check. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Issues
Based on #173, which this PR supersedes, with the review feedback on it addressed.
Description
Summary: Delete large instance subtrees in batches that stay under
jute.maxbuffer, keep the drop atomic for every instance small enough to fit in one batch, clean up after the rare partial drop of a large instance, and never delete an instance re-added while it is being dropped.Why:
dropInstancedeleted the InstanceConfig and the whole/INSTANCES/{instance}subtree in onemulti(). On instances with about 13K MESSAGES the packet was about 3.75 MB; ZK rejected it, the client saw a connection loss, and the default 24hZkClientretry pinned 1,880 Helix REST threads.1.
dropInstance: atomic when it fits, two steps when it doesn'tZKHelixAdmin.dropInstanceruns two steps, each retried on its own.DELETE_RECURSIVE_BATCH_SIZE, 1,000 ops, covering almost every instance), they are deleted in one atomicmulti(), as ondev, so a failed drop leaves nothing behind. Otherwise only the config is deleted, so the controller immediately stops counting the instance (auto maintenance mode budget, delayed rebalance, capacity).HelixConflictException.multi()as a connection loss, whichZkClientresends until its operation timeout instead of failing.NotEmpty,NoNodeand a lost fence (while there is no config) are retried up to 3 times, listing the subtree again each time; other errors fail fast. The precondition fails only when both the config and the subtree are gone, so calling it again finishes a partial drop.2. Batched
ZkClient.deleteRecursivelyThe subtree is walked children first, then deleted in
multi()batches of at most 1,000 ops (about 240 KB). Every caller benefits, includingdropClusterandZkBaseDataAccessor.deleteRecursively.NoNode, for example because a znode was deleted concurrently, or the batch committed but its response was lost and the resend found the znodes gone, that batch is redone one znode at a time and missing znodes are skipped.ZkClientException -> ZkException -> KeeperException.NotEmptyException, the shape callers retry on.ZkClientExceptionlike a failed delete, sodropInstancereports it as aHelixException(REST 400) instead of a 500.3. Fencing against a concurrent re-add
Every leftover subtree delete (step 2 of a drop,
addInstance, the garbage collector) first writes empty data to/INSTANCES/{instance}, which only bumps its data version, then checks that the instance has no config, then deletes the subtree in batches that each start with anOp.checkon that version./INSTANCES/{instance}, so its version no longer matches. The remaining batches fail their check and abort without deleting any of the new znodes.HelixConflictException(REST 409).4. Recovering from a partial drop of a large instance
A large drop that fails between its steps leaves the subtree without a config.
InstanceGarbageCollector(new)ClusterConfig.INSTANCE_GARBAGE_COLLECTION_INTERVAL_MS(new, 1 minute by default; 0 or negative turns it off for the cluster). It finds such subtrees from ZK and deletes them once seen for a minute, so partial drops by any caller are cleaned up, including after a controller failover. Uses the fencedHelixAdmin.deleteDroppedInstanceGarbage(new), so a re-added or live instance is never touched. A failure on one instance does not stop the others.addInstance, RESTPUT /clusters/{c}/instances/{i}, auto-join participantsHelixConflictException(REST 409).dropInstance, RESTDELETEHelixConflictException, REST 409).ClusterSetup.dropInstanceFromCluster(CLI, legacy webapp,swapInstance)deleteDroppedInstanceGarbage, so an instance re-added meanwhile is kept.getInstancesInCluster,getInstancesInClusterWithTagrebalancecannot assign partitions to it.Tests
zookeeper-apiTestRawZkClient:testDeleteRecursivelyLargeSubtree: 2,500 children deleted inmulti()calls of 1000, 1000 and 501 ops.testDeleteRecursivelyDeepSubtree: 1,204 znodes over 3 levels, deleted in calls of 1000 and 204 ops, so a batch boundary falls between levels.testDeleteRecursivelyResumesAfterBatchFailure: the second batch fails, the first stays deleted, and a second call deletes the rest.testDeleteRecursivelyToleratesConcurrentDelete: a znode deleted between the walk and its batch is skipped.testDeleteRecursivelyToleratesResentBatch: a batch resent after it committed does not fail the call.testDeleteRecursivelyNotEmptyKeepsRetryableExceptionShape: a child created concurrently surfaces as the retryable NotEmpty shape.testDeleteRecursivelyWrapsListingFailure: a failure while walking the subtree is wrapped inZkClientException.testDeleteInBatchesStopsWhenFenceChanges,testDeleteInBatchesReturnsWhenFenceIsGone: a fenced delete aborts once the fence znode is recreated, and returns once it is gone.testDeleteRecursivelyOnMissingPathIsNoOp,testDeleteRecursivelySkipsChildDeletedBeforeWalk.helix-coreTestZkHelixAdmin:testDropInstanceIsAtomicUnlessSubtreeExceedsOneBatch: a small instance is deleted in onemulti()that includes the config; a large one deletes the config, then fences the subtree root, then deletes the subtree in batches of at most 1,000 ops that each check the fence first.testDropInstanceKeepsInstanceReAddedDuringDrop: an instance re-added between batches keeps its config and new subtree, and the drop reports a conflict.testDropInstanceRetryKeepsInstanceReAddedAfterConfigDelete: a large drop loses its fence to another deleter, the instance is re-added before the retry, and the retry keeps it and reports a conflict instead of dropping it again.testDropInstanceWithLargeMessageSubtree,testDropInstanceWithDeepSubtreeShape: end to end.testDropInstanceResumesAfterPartialDelete: config gone, subtree left, drop completes.testDropInstanceRetriesOnNoNode,testDropInstanceGivesUpAfterRepeatedNoNode,testDropInstanceCompletesWhenZnodeDeletedMidDrop: NoNode on the atomic delete is retried, up to 3 times.testDropInstanceFailsFastOnNonRetryableError,testDropInstanceReportsSubtreeListingFailure: other errors are not retried and surface asHelixException.testAddInstanceCleansLeftoverSubtree,testAddInstanceRejectsLeftoverSubtreeOfLiveInstance,testAddInstanceSucceedsAfterDropClearsLeftoverSubtree.testInstanceListsExcludeLeftoverSubtree: instance lists omit a leftover instance.testZkHelixAdmin: the NotEmpty retry mock now targetsdeleteRecursivelyAtomic, with the same 3 retry semantics asdev.helix-coreTestInstanceGarbageCollector(real ZK):testDeletesLeftoverSubtreeAfterGracePeriod: a leftover is deleted only after a full grace period; configured and live instances are kept.testKeepsInstanceReAddedOrLiveSinceFirstSeen: an instance re-added or come live after it was first seen is not deleted.testOnlyLeaderCollects: a non-leader deletes nothing and forgets what it has seen.testLeaderControllerSchedulesCollection: a started leader controller schedules the collection without any cluster event.testIntervalFromClusterConfig: 0 or negative turns the collection off; turned back on, a leftover still waits a full grace period and runs follow the configured interval.testFailureOnOneLeftoverDoesNotStopOthers: an unexpected error deleting one leftover does not stop the others from being collected.helix-coreTestClusterSetup:testDropInstanceFromClusterResumesPartialDrop.testDropInstanceFromClusterKeepsInstanceReAddedDuringPartialDrop: an instance re-added before ClusterSetup finishes a partial drop is kept, with a conflict.helix-restTestPerInstanceAccessor:testDeleteInstanceResumesPartialDrop: DELETE finishes a partial drop (200), then returns 404; PUT cleans a leftover and adds the instance (200); PUT on a live instance's leftover returns 409.testDeleteInstanceReturnsConflictWhenReAddedDuringDrop: DELETE returns 409 when the drop finds the instance re-added.The batching, NoNode, atomic-choice, fencing, re-add, 409, ClusterSetup and
addInstancetests fail against the previous code and pass with this change.Run with JDK 11 against this branch, after
mvn -pl helix-rest -am clean install -DskipTests:Changes that Break Backward Compatibility (Optional)
My PR contains changes that break backward compatibility or previous assumptions for certain methods or API. They include:
REST
DELETE /clusters/{cluster}/instances/{instance}returns 404 instead of 400 when the instance does not exist, 409 when the instance is re-added while it is being dropped, and succeeds when only the subtree remains.dropInstanceof an instance whose config and subtree exceed one batch is no longer atomic. A failure between the steps leaves the subtree without a config until a caller or the controller's garbage collector finishes the drop.dropInstancefails with "does not exist in config" only when both the config and the subtree are gone; the separate "does not exist in instances" error is removed.addInstancedeletes a leftover subtree without an InstanceConfig before adding, instead of creating the config on top of it; it throwsHelixConflictException(REST 409 instead of 500) if that instance is live.getInstancesInClusterno longer lists instances without an InstanceConfig, andgetInstancesInClusterWithTagskips them instead of throwingIllegalStateException.dropInstancethrowsHelixConflictException(aHelixException) when the instance is re-added while it is being dropped, and keeps the re-added instance.ClusterSetup.dropInstanceFromClusterfinishes a partial drop instead of throwing "does not exist".The leader controller runs a new periodic task,
InstanceGarbageCollector, that lists/INSTANCESonce a minute by default. NewClusterConfigfieldINSTANCE_GARBAGE_COLLECTION_INTERVAL_MSsets the interval; 0 or negative turns it off.New public API:
HelixAdmin.deleteDroppedInstanceGarbage(default method, implemented byZKHelixAdmin), andgetSubtreeChildrenFirst/deleteInBatchesonRealmAwareZkClientand its implementations.Documentation (Optional)
N/A. The
deleteRecursivelyjavadoc documents the batching, atomicity, and exception shapes.Commits
Code Quality
(helix-style-intellij.xml if IntelliJ IDE is used)