Skip to content

Close tracers built by debugger tests (quick fix) - #13040

Draft
dougqh wants to merge 1 commit into
masterfrom
dougqh/debugger-tests-close-tracers
Draft

dougqh wants to merge 1 commit into
masterfrom
dougqh/debugger-tests-close-tracers

Conversation

@dougqh

@dougqh dougqh commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

Debugger tests now close the CoreTracers they build.

Motivation

:dd-java-agent:agent-debugger:test runs out of its 512 MB heap on master.

  • What a heap dump showed: 46 PayloadDispatcherImpl instances retained 225 MB, 65% of the heap, about 5.2 MB each.
  • What holds them: live TraceProcessingWorker serializer threads.
  • Why: eight test classes call CoreTracer.builder().build(), which uses the default agent writer, and never close the tracer. The writer's thread stays alive and pins its msgpack buffer, created on the first trace written. ExceptionProbeInstrumentationTest builds one per test in @BeforeEach, and Add coordinated sampling for snapshot probes #12452 added CoordinatedSamplingTest with nine more call sites.

The fix:

  • Each test keeps a reference to the tracer it builds and closes it in teardown.
  • For CapturingTestBase subclasses, that's a shared field, closed in CapturingTestBase.after().
  • LogProbeTest builds several per test, so it closes them in a list @AfterEach.

The test task now completes: 950 tests, 15 skipped, no failures, no OOM.

Additional Notes

Test-only change.

🤖 Generated with Claude Code

Debugger tests built CoreTracers with the default agent writer and never
closed them. Each one kept a serializer thread alive, pinning that
writer's 5 MB payload buffer, so the test JVM accumulated them until
agent-debugger:test ran out of its 512 MB heap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh dougqh added tag: no release notes Changes to exclude from release notes comp: debugger Dynamic Instrumentation tag: ai generated Largely based on code generated by an AI or LLM type: bug labels Oct 23, 2026
@datadog-official

This comment has been minimized.

@pr-commenter

pr-commenter Bot commented Oct 24, 2026

Copy link
Copy Markdown

Debugger benchmarks

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
ci_job_date 1792799525 1792799853
end_time 2026-10-23T23:53:36 2026-10-23T23:59:04
git_branch master dougqh/debugger-tests-close-tracers
git_commit_sha 13d72f6 d3ba02a
start_time 2026-10-23T23:52:06 2026-10-23T23:57:34
See matching parameters
Baseline Candidate
ci_job_id 2172854612 2172854612
ci_pipeline_id 147248372 147248372
cpu_model Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
git_commit_date 1792798792 1792798792

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 6 unstable metrics.

See unchanged results
scenario Δ mean agg_http_req_duration_min Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p75 Δ mean agg_http_req_duration_p99 Δ mean throughput
scenario:noprobe unstable
[-26.101µs; +22.079µs] or [-7.762%; +6.566%]
unstable
[-33178.133ns; +34497.880ns] or [-8.965%; +9.322%]
unstable
[-43155.422ns; +44099.367ns] or [-11.214%; +11.459%]
unstable
[-52.532µs; +181.403µs] or [-4.810%; +16.609%]
same
scenario:basic same same same unstable
[+197.595µs; +476.904µs] or [+23.122%; +55.805%]
unstable
[-132.690op/s; +132.690op/s] or [-5.971%; +5.971%]
scenario:loop same same same same same
Request duration reports for reports
gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (370.086 µs) : 331, 410
.   : milestone, 370,
basic (343.305 µs) : 337, 349
.   : milestone, 343,
loop (8.116 ms) : 8052, 8180
.   : milestone, 8116,
section candidate
noprobe (370.746 µs) : 348, 393
.   : milestone, 371,
basic (345.424 µs) : 338, 353
.   : milestone, 345,
loop (8.123 ms) : 8060, 8187
.   : milestone, 8123,
Loading
  • baseline results
Scenario Request median duration [CI 0.99]
noprobe 370.086 µs [330.501 µs, 409.672 µs]
basic 343.305 µs [337.261 µs, 349.35 µs]
loop 8.116 ms [8.052 ms, 8.18 ms]
  • candidate results
Scenario Request median duration [CI 0.99]
noprobe 370.746 µs [348.45 µs, 393.043 µs]
basic 345.424 µs [337.773 µs, 353.076 µs]
loop 8.123 ms [8.06 ms, 8.187 ms]

@dd-octo-sts

dd-octo-sts Bot commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.95 s 13.88 s [-0.3%; +1.2%] (no difference)
startup:insecure-bank:tracing:Agent 12.94 s 12.98 s [-1.0%; +0.5%] (no difference)
startup:petclinic:appsec:Agent 17.16 s 16.98 s [+0.1%; +2.0%] (maybe worse)
startup:petclinic:iast:Agent 16.96 s 17.00 s [-1.0%; +0.5%] (no difference)
startup:petclinic:profiling:Agent 16.54 s 16.12 s [-2.0%; +7.1%] (no difference)
startup:petclinic:sca:Agent 17.13 s 16.89 s [+0.5%; +2.3%] (maybe worse)
startup:petclinic:tracing:Agent 16.04 s 16.15 s [-1.6%; +0.3%] (no difference)

Commit: d3ba02ac · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: debugger Dynamic Instrumentation tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: bug

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant