Skip to content

Stop the management mode pipeline on shutdown - #443

Open
LZD-PratyushBhatt wants to merge 1 commit into
devfrom
lzd/stop-management-pipeline-on-shutdown
Open

LZD-PratyushBhatt wants to merge 1 commit into
devfrom
lzd/stop-management-pipeline-on-shutdown

Conversation

@LZD-PratyushBhatt

Copy link
Copy Markdown
Collaborator

GenericHelixController starts three pipeline threads: default, task and management mode. shutdown() stops only the first two. The management mode thread is an inner class of the controller, so after shutdown it still keeps the whole controller reachable, including its HelixManager, ZkClients, monitors and cached cluster state.

A distributed controller builds a new GenericHelixController every time it regains leadership of a cluster, so each lost leadership leaks one, and the heap only comes back when the process restarts. In a stress run on a WAGED cluster with about 373k replicas, each handover held on to close to 1 GB.

Fix: stop the management mode pipeline in shutdown() the same way as the other two.

Testing Done

TestGenericHelixControllerThreading#testShutdownStopsEveryPipelineThread fails on dev on the management mode thread and passes with the fix.
Freeze mode, management mode, distributed election, leadership change and session handling tests pass (27 tests).

GenericHelixController starts a default, a task and a management mode
pipeline thread, but shutdown() stopped only the first two. The management
mode thread is an inner class of the controller, so it kept the whole
controller reachable after shutdown: its HelixManager, ZkClients, monitors
and cached cluster state. A distributed controller builds a new
GenericHelixController each time it regains a cluster, so every lost
leadership leaked one.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant