Skip to content

fix: prevent memory leak by reusing gRPC conn to kubelet pod-resource… - #705

Open
AshwinRamaniPsg wants to merge 1 commit into
NVIDIA:mainfrom
AshwinRamaniPsg:main
Open

fix: prevent memory leak by reusing gRPC conn to kubelet pod-resource…#705
AshwinRamaniPsg wants to merge 1 commit into
NVIDIA:mainfrom
AshwinRamaniPsg:main

Conversation

@AshwinRamaniPsg

Copy link
Copy Markdown

…s API

  • Add persistent grpcConn to PodMapper, reused across scrapes instead of creating a new *grpc.ClientConn on every /metrics request (issue dcgm-exporter 4.4.2 memory leak causing periodic OOM kills #702)
  • Reset cached conn on RPC error to handle kubelet restarts
  • Move resolver.SetDefaultScheme to one-time init in NewPodMapper
  • Clone labels map per entity in expCollector and gpuHealthStatusCollector to prevent cross-GPU label pollution and extra GC pressure
  • Add tests: connection reuse, reconnect after shutdown, Stop() cleanup

@AshwinRamaniPsg

AshwinRamaniPsg commented Jul 2, 2026

Copy link
Copy Markdown
Author

@nccurry would you be able to help review this PR please

@rvatkar

rvatkar commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Could you please rebase in PR

@AshwinRamaniPsg

Copy link
Copy Markdown
Author

Could you please rebase in PR

Done

…s API

- Add persistent grpcConn to PodMapper, reused across scrapes instead of
  creating a new *grpc.ClientConn on every /metrics request (issue NVIDIA#702)
- Reset cached conn on RPC error to handle kubelet restarts
- Move resolver.SetDefaultScheme to one-time init in NewPodMapper
- Clone labels map per entity in expCollector and gpuHealthStatusCollector
  to prevent cross-GPU label pollution and extra GC pressure
- Add tests: connection reuse, reconnect after shutdown, Stop() cleanup
@AshwinRamaniPsg

Copy link
Copy Markdown
Author

@rvatkar can you please help review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants