Skip to content

fix: memory leak when profiling is enabled#5133

Merged
jamescrosswell merged 4 commits intomainfrom
sdk-version-bag-5113
Apr 15, 2026
Merged

fix: memory leak when profiling is enabled#5133
jamescrosswell merged 4 commits intomainfrom
sdk-version-bag-5113

Conversation

@jamescrosswell
Copy link
Copy Markdown
Collaborator

@jamescrosswell jamescrosswell commented Apr 14, 2026

fixes: #5113
fixes: #4721

Fixes memory leaks caused by ConcurrentBag and ConcurrentQueue.

Why ConcurrentBag leaks

ConcurrentBag is backed by a ThreadLocal — internally it keeps a per-thread linked list. Every thread that ever touches the bag allocates a ThreadLocalList node that stays attached to the bag for the bag's lifetime, even after that thread has died. In a high-throughput app (profiler + ASP.NET Core thread pool), each bag instance accumulates stranded per-thread slots, and each SdkVersion instance carries that weight.

Why the profiler amplifies it

Every SentryEvent, SentryTransaction, Scope, and TransactionTracer allocates its own SdkVersion → two fresh ConcurrentBag instances. AddPackage gets called on most of these from multiple callsites (SentryMiddleware.cs:252, Enricher.cs:72, Scope.cs:527). The profiler drives the event/transaction throughput way up, so both the count of bags and the per-bag thread-local overhead explode.

There's also one hotter target: NoOpTransaction.cs:14 returns SdkVersion.Instance (the singleton) as its Sdk. Any code path that ends up calling AddPackage on a NoOpTransaction's Sdk writes into that singleton bag forever.

The fix pattern

Same as #3432: replace ConcurrentBag with a simpler lock-backed list (as suggested in #4721).

Testing

Memory does keep growing in the process but this seems to be just because I have loads of memory so the GC doesn't bother kicking in. If I force a GC then memory is released again:
image

As far as I can tell then, there are no remaining memory leaks in the minimal repro that @kanadaj provided.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 14, 2026

Semver Impact of This PR

None (no version bump detected)

📋 Changelog Preview

This is how your changes will appear in the changelog.
Entries from this PR are highlighted with a left border (blockquote style).


Fixes 🐛

  • fix: memory leak when profiling is enabled by jamescrosswell in #5133
  • fix: prevent redundant native exceptions on Android/Mono by jpnurmi in #4676
    • Note: opt in by setting options.Native.ExperimentalOptions.SignalHandlerStrategy to Sentry.Android.SignalHandlerStrategy.ChainAtStart

Dependencies ⬆️

Deps

  • chore(deps): update Cocoa SDK to v9.10.0 by github-actions in #5132
  • chore(deps): update Cocoa SDK to v9.9.0 by github-actions in #5115
  • chore(deps): update Java SDK to v8.38.0 by github-actions in #5124

🤖 This preview updates automatically when you update the PR.

@codecov
Copy link
Copy Markdown

codecov bot commented Apr 14, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.03%. Comparing base (ec94d54) to head (1bb8430).
⚠️ Report is 14 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5133      +/-   ##
==========================================
+ Coverage   73.99%   74.03%   +0.03%     
==========================================
  Files         499      500       +1     
  Lines       18067    18090      +23     
  Branches     3520     3520              
==========================================
+ Hits        13369    13393      +24     
+ Misses       3838     3837       -1     
  Partials      860      860              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jamescrosswell and others added 3 commits April 14, 2026 22:10
This was passing accidentally before because ConcurrentBag typically
followed a LIFO model for adding/removing items.

The SQL listener test had a bug that the LIFO mask would hide. The test
did `OrderByDescending(x => x.StartTimestamp).First()` to get the
"newest" span, but this fails whenever two spans share the same
timestamp — which is common on Windows due to ~15ms clock resolution.
Making ConcurrentBagLite LIFO would make the test pass again
accidentally, because the newer span would happen to enumerate first and
survive the stable sort. But the brittleness is still there — if the
spans ever get equal timestamps in a different order (e.g., a background
thread adds one), the test breaks again.
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SQL listener test had a bug that the LIFO mask would hide. The test did OrderByDescending(x => x.StartTimestamp).First() to get the "newest" span, but this fails whenever two spans share the same
timestamp — which is common on Windows x64 due to ~15ms clock resolution. Making ConcurrentBagLite LIFO would make the test pass again accidentally, because the newer span would happen to enumerate first and
survive the stable sort... so changing/fixing the test instead.

@jamescrosswell jamescrosswell marked this pull request as ready for review April 15, 2026 06:46
@jamescrosswell jamescrosswell requested a review from vaind April 15, 2026 08:09

public void Add(T item)
{
lock (_items)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth checking whether we have a hotspot where add(IEnumearble<T> items) would be useful.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call... we have these:

foreach (var attachment in Attachments)
{
// Set the attachment directly to avoid triggering a scope sync
other._attachments.Add(attachment);
}

foreach (var processor in EventProcessors)
{
clone.EventProcessors.Add(processor);
}
foreach (var processor in TransactionProcessors)
{
clone.TransactionProcessors.Add(processor);
}
foreach (var processor in ExceptionProcessors)
{
clone.ExceptionProcessors.Add(processor);
}

foreach (var processor in processors)
{
ExceptionProcessors.Add(processor);
}

foreach (var processor in processors)
{
EventProcessors.Add(processor);
}

foreach (var processor in processors)
{
TransactionProcessors.Add(processor);
}

I don't think I'd describe any of those as a hot path though... with the exception of the first one (attachments) it's all one off stuff that happens at init for a very constrained number of items - and even attachments is likely to be only a handful of items.

I think I'll leave it for now then, in the interests of expediting the fix to the memory leak.

@jamescrosswell jamescrosswell merged commit e99c3d1 into main Apr 15, 2026
48 checks passed
@jamescrosswell jamescrosswell deleted the sdk-version-bag-5113 branch April 15, 2026 23:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sentry profiler seems to be leaking memory Replace applicable ConcurrentBag usages with custom lightweight thread-safe collection

2 participants