Skip to content

fix(shared-runtime): discard inherited MicroVM runtime data - #2478

Open
litianningdatadog wants to merge 3 commits into
mainfrom
tianning.li/buffer-reset-upon-microvm-start
Open

fix(shared-runtime): discard inherited MicroVM runtime data#2478
litianningdatadog wants to merge 3 commits into
mainfrom
tianning.li/buffer-reset-upon-microvm-start

Conversation

@litianningdatadog

@litianningdatadog litianningdatadog commented Sep 8, 2026

Copy link
Copy Markdown

Why

AWS Lambda MicroVM snapshots can restore a process with trace, stats, and telemetry data accumulated under a previous runtime identity. This adds a no-flush identity-refresh path so /run callers can discard that inherited exporter state before creating a replacement exporter.

It does not eliminate the small window between process restore and completion of the /run refresh. Data emitted in that window may still carry the snapshotted runtime ID. The RFC accepts that MVP gap; closing it requires a pre-snapshot worker hold and the associated lifecycle and accounting changes.

How

Adds a discard-only exporter teardown for the MicroVM identity-refresh path.

TraceExporter::shutdown_without_flush() races each worker's in-flight async run against cancellation rather than waiting for normal pause/shutdown behavior. It does not preempt synchronous work already running inside a worker or remove the pre-/run race. It removes the worker through Worker::discard(). discard() defaults to reset(), but lets a worker override cleanup that must differ from its fork-continuation reset(); the trace-buffer worker uses this to close its channel instead of reopening it. That worker is not wired into an exporter yet, but the generic discard contract needs to handle it correctly before it is. The exporter also clears the global agent-info cache so the replacement exporter does not skip its first /info fetch. The FFI exposes this as ddog_trace_exporter_free_without_flush.

This is intentionally scoped to the identity-refresh API. Existing WorkerHandle::stop, runtime shutdown, and fork lifecycle behavior are unchanged: pause() still waits for an in-flight run to finish, and reset() still reopens a worker for continued use. Callers use discard in the MicroVM startup identity-refresh path, not concurrently with ordinary shutdown or a real fork.

References

Verification

  • cargo +nightly-2026-07-26 fmt --all -- --check
  • cargo check -p libdd-shared-runtime -p libdd-data-pipeline -p libdd-data-pipeline-ffi
  • cargo test -p libdd-shared-runtime
  • cargo test -p libdd-data-pipeline test_shutdown_without_flush
  • cargo test -p libdd-data-pipeline test_worker_discard
  • cargo test -p libdd-data-pipeline-ffi exporter_free_without_flush
  • cargo +stable clippy -p libdd-shared-runtime --all-targets -- -D warnings

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

📚 Documentation Check Results

⚠️ 2912 documentation warning(s) found

📦 libdd-data-pipeline-ffi - 1405 warning(s)

📦 libdd-data-pipeline - 1294 warning(s)

📦 libdd-shared-runtime - 213 warning(s)


Updated: 2026-09-09 16:33:40 UTC | Commit: c23a9f2 | missing-docs job results

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

🔒 Cargo Deny Results

No issues found!

📦 libdd-data-pipeline-ffi - ✅ No issues

📦 libdd-data-pipeline - ✅ No issues

📦 libdd-shared-runtime - ✅ No issues


Updated: 2026-09-09 16:34:49 UTC | Commit: c23a9f2 | dependency-check job results

@datadog-official

datadog-official Bot commented Sep 8, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 89.51%
Overall Coverage: 77.70% (+0.05%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 817aa89 | Docs | View more details | Give us feedback!

@pr-commenter

pr-commenter Bot commented Sep 8, 2026

Copy link
Copy Markdown

Benchmarks

Comparison

Benchmark execution time: 2026-09-09 16:43:55

Comparing candidate commit 817aa89 in PR branch tianning.li/buffer-reset-upon-microvm-start with baseline commit 643ea5e in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 25 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Benchmark execution time: 2026-09-09 16:43:31

Comparing candidate commit 817aa89 in PR branch tianning.li/buffer-reset-upon-microvm-start with baseline commit 643ea5e in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Candidate

Candidate benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz 817aa89 1788971451 tianning.li/buffer-reset-upon-microvm-start
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
trace_buffer/1_senders/10us_delay execution_time 59.992ms 60.084ms ± 0.037ms 60.073ms ± 0.013ms 60.093ms 60.156ms 60.218ms 60.267ms 0.32% 1.862 4.928 0.06% 0.003ms 1 200
trace_buffer/1_senders/10us_delay throughput 14933.563op/s 14979.007op/s ± 9.300op/s 14981.765op/s ± 3.199op/s 14983.606op/s 14988.911op/s 14990.016op/s 15001.922op/s 0.13% -1.856 4.901 0.06% 0.658op/s 1 200
trace_buffer/1_senders/1us_delay execution_time 50.139ms 50.366ms ± 0.163ms 50.332ms ± 0.102ms 50.453ms 50.638ms 50.809ms 51.207ms 1.74% 1.452 3.635 0.32% 0.012ms 1 200
trace_buffer/1_senders/1us_delay throughput 17575.751op/s 17869.414op/s ± 57.738op/s 17881.407op/s ± 36.356op/s 17909.920op/s 17942.015op/s 17948.005op/s 17950.120op/s 0.38% -1.418 3.451 0.32% 4.083op/s 1 200
trace_buffer/1_senders/no_delay execution_time 351.686µs 360.000µs ± 7.023µs 359.327µs ± 1.354µs 360.724µs 364.838µs 376.046µs 444.608µs 23.73% 8.981 104.002 1.95% 0.497µs 1 200
trace_buffer/1_senders/no_delay throughput 2024253.474op/s 2500813.220op/s ± 42025.508op/s 2504680.857op/s ± 9418.390op/s 2513771.771op/s 2543880.887op/s 2554623.973op/s 2559098.656op/s 2.17% -7.584 81.517 1.68% 2971.652op/s 1 200
trace_buffer/2_senders/10us_delay execution_time 60.071ms 60.142ms ± 0.052ms 60.132ms ± 0.031ms 60.160ms 60.261ms 60.282ms 60.287ms 0.26% 1.107 0.538 0.09% 0.004ms 1 200
trace_buffer/2_senders/10us_delay throughput 29857.225op/s 29929.081op/s ± 25.944op/s 29934.245op/s ± 15.208op/s 29949.526op/s 29958.403op/s 29962.105op/s 29964.355op/s 0.10% -1.103 0.530 0.09% 1.834op/s 1 200
trace_buffer/2_senders/1us_delay execution_time 50.642ms 51.078ms ± 0.169ms 51.086ms ± 0.113ms 51.187ms 51.351ms 51.514ms 51.557ms 0.92% 0.174 -0.017 0.33% 0.012ms 1 200
trace_buffer/2_senders/1us_delay throughput 34912.925op/s 35240.401op/s ± 116.468op/s 35234.925op/s ± 77.899op/s 35314.233op/s 35429.513op/s 35477.016op/s 35543.504op/s 0.88% -0.155 -0.031 0.33% 8.236op/s 1 200
trace_buffer/2_senders/no_delay execution_time 1.483ms 1.604ms ± 0.041ms 1.601ms ± 0.025ms 1.629ms 1.682ms 1.712ms 1.728ms 7.94% 0.243 0.314 2.57% 0.003ms 1 200
trace_buffer/2_senders/no_delay throughput 1041366.879op/s 1122851.764op/s ± 28803.856op/s 1124032.598op/s ± 18089.546op/s 1141542.551op/s 1167590.917op/s 1193891.662op/s 1213391.652op/s 7.95% -0.068 0.315 2.56% 2036.740op/s 1 200
trace_buffer/4_senders/10us_delay execution_time 60.134ms 60.186ms ± 0.042ms 60.173ms ± 0.022ms 60.202ms 60.278ms 60.314ms 60.385ms 0.35% 1.725 3.573 0.07% 0.003ms 1 200
trace_buffer/4_senders/10us_delay throughput 59617.191op/s 59814.892op/s ± 41.764op/s 59827.213op/s ± 21.734op/s 59843.205op/s 59858.880op/s 59865.343op/s 59866.566op/s 0.07% -1.720 3.546 0.07% 2.953op/s 1 200
trace_buffer/4_senders/1us_delay execution_time 51.081ms 51.322ms ± 0.158ms 51.286ms ± 0.071ms 51.375ms 51.651ms 51.923ms 51.958ms 1.31% 1.600 3.134 0.31% 0.011ms 1 200
trace_buffer/4_senders/1us_delay throughput 69287.073op/s 70146.039op/s ± 214.314op/s 70195.184op/s ± 96.580op/s 70267.364op/s 70396.607op/s 70454.893op/s 70476.460op/s 0.40% -1.576 3.037 0.30% 15.154op/s 1 200
trace_buffer/4_senders/no_delay execution_time 3.476ms 3.584ms ± 0.046ms 3.583ms ± 0.023ms 3.605ms 3.639ms 3.665ms 4.023ms 12.27% 4.128 38.476 1.29% 0.003ms 1 200
trace_buffer/4_senders/no_delay throughput 894822.156op/s 1004680.646op/s ± 12420.706op/s 1004628.579op/s ± 6544.462op/s 1011786.973op/s 1021321.618op/s 1029122.847op/s 1035697.978op/s 3.09% -3.302 29.106 1.23% 878.277op/s 1 200
trace_buffer/8_senders/10us_delay execution_time 60.248ms 60.336ms ± 0.066ms 60.319ms ± 0.037ms 60.378ms 60.455ms 60.540ms 60.651ms 0.55% 1.328 2.269 0.11% 0.005ms 1 200
trace_buffer/8_senders/10us_delay throughput 118712.296op/s 119331.401op/s ± 130.718op/s 119365.042op/s ± 73.074op/s 119432.507op/s 119471.209op/s 119498.159op/s 119505.228op/s 0.12% -1.320 2.227 0.11% 9.243op/s 1 200
trace_buffer/8_senders/1us_delay execution_time 51.251ms 51.672ms ± 0.320ms 51.540ms ± 0.143ms 51.825ms 52.332ms 52.480ms 52.633ms 2.12% 1.069 0.017 0.62% 0.023ms 1 200
trace_buffer/8_senders/1us_delay throughput 136796.223op/s 139345.558op/s ± 856.360op/s 139698.363op/s ± 388.013op/s 139983.038op/s 140201.853op/s 140299.648op/s 140484.639op/s 0.56% -1.052 -0.024 0.61% 60.554op/s 1 200
trace_buffer/8_senders/no_delay execution_time 6.976ms 7.115ms ± 0.082ms 7.099ms ± 0.055ms 7.169ms 7.279ms 7.326ms 7.337ms 3.35% 0.697 -0.312 1.15% 0.006ms 1 200
trace_buffer/8_senders/no_delay throughput 981348.785op/s 1012142.549op/s ± 11541.799op/s 1014216.284op/s ± 7821.056op/s 1021111.408op/s 1027303.664op/s 1029223.962op/s 1032170.410op/s 1.77% -0.656 -0.382 1.14% 816.128op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
trace_buffer/1_senders/10us_delay execution_time [60.079ms; 60.089ms] or [-0.009%; +0.009%] None None None
trace_buffer/1_senders/10us_delay throughput [14977.719op/s; 14980.296op/s] or [-0.009%; +0.009%] None None None
trace_buffer/1_senders/1us_delay execution_time [50.343ms; 50.389ms] or [-0.045%; +0.045%] None None None
trace_buffer/1_senders/1us_delay throughput [17861.412op/s; 17877.416op/s] or [-0.045%; +0.045%] None None None
trace_buffer/1_senders/no_delay execution_time [359.027µs; 360.973µs] or [-0.270%; +0.270%] None None None
trace_buffer/1_senders/no_delay throughput [2494988.889op/s; 2506637.551op/s] or [-0.233%; +0.233%] None None None
trace_buffer/2_senders/10us_delay execution_time [60.135ms; 60.149ms] or [-0.012%; +0.012%] None None None
trace_buffer/2_senders/10us_delay throughput [29925.485op/s; 29932.676op/s] or [-0.012%; +0.012%] None None None
trace_buffer/2_senders/1us_delay execution_time [51.055ms; 51.102ms] or [-0.046%; +0.046%] None None None
trace_buffer/2_senders/1us_delay throughput [35224.260op/s; 35256.542op/s] or [-0.046%; +0.046%] None None None
trace_buffer/2_senders/no_delay execution_time [1.598ms; 1.610ms] or [-0.357%; +0.357%] None None None
trace_buffer/2_senders/no_delay throughput [1118859.826op/s; 1126843.701op/s] or [-0.356%; +0.356%] None None None
trace_buffer/4_senders/10us_delay execution_time [60.180ms; 60.192ms] or [-0.010%; +0.010%] None None None
trace_buffer/4_senders/10us_delay throughput [59809.104op/s; 59820.680op/s] or [-0.010%; +0.010%] None None None
trace_buffer/4_senders/1us_delay execution_time [51.300ms; 51.344ms] or [-0.043%; +0.043%] None None None
trace_buffer/4_senders/1us_delay throughput [70116.337op/s; 70175.741op/s] or [-0.042%; +0.042%] None None None
trace_buffer/4_senders/no_delay execution_time [3.577ms; 3.590ms] or [-0.180%; +0.180%] None None None
trace_buffer/4_senders/no_delay throughput [1002959.256op/s; 1006402.037op/s] or [-0.171%; +0.171%] None None None
trace_buffer/8_senders/10us_delay execution_time [60.327ms; 60.345ms] or [-0.015%; +0.015%] None None None
trace_buffer/8_senders/10us_delay throughput [119313.285op/s; 119349.517op/s] or [-0.015%; +0.015%] None None None
trace_buffer/8_senders/1us_delay execution_time [51.628ms; 51.716ms] or [-0.086%; +0.086%] None None None
trace_buffer/8_senders/1us_delay throughput [139226.875op/s; 139464.242op/s] or [-0.085%; +0.085%] None None None
trace_buffer/8_senders/no_delay execution_time [7.103ms; 7.126ms] or [-0.159%; +0.159%] None None None
trace_buffer/8_senders/no_delay throughput [1010542.966op/s; 1013742.131op/s] or [-0.158%; +0.158%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz 817aa89 1788971451 tianning.li/buffer-reset-upon-microvm-start
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
concentrator/add_spans_to_concentrator execution_time 9.743ms 9.771ms ± 0.018ms 9.765ms ± 0.010ms 9.780ms 9.807ms 9.822ms 9.846ms 0.83% 1.241 1.701 0.19% 0.001ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
concentrator/add_spans_to_concentrator execution_time [9.768ms; 9.774ms] or [-0.026%; +0.026%] None None None

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz 817aa89 1788971451 tianning.li/buffer-reset-upon-microvm-start
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
receiver_entry_point/report/2644 execution_time 6.861ms 6.953ms ± 0.027ms 6.956ms ± 0.016ms 6.971ms 6.992ms 7.011ms 7.025ms 1.00% -0.656 1.118 0.38% 0.002ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
receiver_entry_point/report/2644 execution_time [6.950ms; 6.957ms] or [-0.053%; +0.053%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz 817aa89 1788971451 tianning.li/buffer-reset-upon-microvm-start
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
two way interface execution_time 20.767µs 21.324µs ± 0.267µs 21.296µs ± 0.158µs 21.470µs 21.769µs 22.061µs 22.647µs 6.34% 1.227 3.512 1.25% 0.019µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
two way interface execution_time [21.287µs; 21.361µs] or [-0.173%; +0.173%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz 817aa89 1788971451 tianning.li/buffer-reset-upon-microvm-start
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
write only interface execution_time 2.052µs 2.076µs ± 0.010µs 2.074µs ± 0.007µs 2.084µs 2.093µs 2.097µs 2.102µs 1.33% 0.351 -0.545 0.46% 0.001µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
write only interface execution_time [2.075µs; 2.078µs] or [-0.063%; +0.063%] None None None

Baseline

Baseline benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz 643ea5e 1788965317 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
trace_buffer/1_senders/10us_delay execution_time 59.975ms 60.086ms ± 0.039ms 60.076ms ± 0.015ms 60.100ms 60.161ms 60.230ms 60.263ms 0.31% 1.650 4.348 0.07% 0.003ms 1 200
trace_buffer/1_senders/10us_delay throughput 14934.581op/s 14978.541op/s ± 9.807op/s 14980.971op/s ± 3.838op/s 14983.834op/s 14988.938op/s 14991.501op/s 15006.294op/s 0.17% -1.642 4.323 0.07% 0.693op/s 1 200
trace_buffer/1_senders/1us_delay execution_time 50.094ms 50.362ms ± 0.176ms 50.326ms ± 0.079ms 50.410ms 50.686ms 50.993ms 51.252ms 1.84% 1.959 5.628 0.35% 0.012ms 1 200
trace_buffer/1_senders/1us_delay throughput 17560.406op/s 17870.683op/s ± 62.001op/s 17883.410op/s ± 28.032op/s 17908.190op/s 17938.710op/s 17957.601op/s 17966.096op/s 0.46% -1.920 5.412 0.35% 4.384op/s 1 200
trace_buffer/1_senders/no_delay execution_time 349.807µs 357.299µs ± 6.918µs 356.243µs ± 2.530µs 358.908µs 365.176µs 370.583µs 436.165µs 22.43% 7.631 82.809 1.93% 0.489µs 1 200
trace_buffer/1_senders/no_delay throughput 2063437.054op/s 2519727.016op/s ± 43044.703op/s 2526364.600op/s ± 18039.411op/s 2542454.704op/s 2559385.470op/s 2566922.093op/s 2572844.797op/s 1.84% -6.229 61.536 1.70% 3043.720op/s 1 200
trace_buffer/2_senders/10us_delay execution_time 60.071ms 60.126ms ± 0.042ms 60.113ms ± 0.021ms 60.143ms 60.224ms 60.249ms 60.251ms 0.23% 1.327 1.154 0.07% 0.003ms 1 200
trace_buffer/2_senders/10us_delay throughput 29875.199op/s 29937.131op/s ± 20.754op/s 29943.730op/s ± 10.257op/s 29950.886op/s 29959.916op/s 29962.015op/s 29964.533op/s 0.07% -1.324 1.146 0.07% 1.467op/s 1 200
trace_buffer/2_senders/1us_delay execution_time 50.654ms 51.073ms ± 0.136ms 51.083ms ± 0.092ms 51.169ms 51.271ms 51.342ms 51.466ms 0.75% -0.274 0.183 0.26% 0.010ms 1 200
trace_buffer/2_senders/1us_delay throughput 34974.553op/s 35243.683op/s ± 93.577op/s 35236.837op/s ± 63.674op/s 35305.352op/s 35392.417op/s 35475.073op/s 35535.129op/s 0.85% 0.291 0.196 0.26% 6.617op/s 1 200
trace_buffer/2_senders/no_delay execution_time 1.581ms 1.674ms ± 0.038ms 1.673ms ± 0.026ms 1.699ms 1.729ms 1.750ms 1.898ms 13.46% 0.966 4.956 2.25% 0.003ms 1 200
trace_buffer/2_senders/no_delay throughput 948369.109op/s 1075516.247op/s ± 23861.867op/s 1076044.297op/s ± 16500.031op/s 1092685.725op/s 1107448.591op/s 1130602.224op/s 1138605.429op/s 5.81% -0.611 3.046 2.21% 1687.289op/s 1 200
trace_buffer/4_senders/10us_delay execution_time 60.133ms 60.192ms ± 0.041ms 60.180ms ± 0.020ms 60.209ms 60.278ms 60.305ms 60.315ms 0.22% 1.125 0.624 0.07% 0.003ms 1 200
trace_buffer/4_senders/10us_delay throughput 59686.665op/s 59808.525op/s ± 40.426op/s 59820.540op/s ± 20.052op/s 59836.963op/s 59855.978op/s 59860.057op/s 59867.175op/s 0.08% -1.122 0.617 0.07% 2.859op/s 1 200
trace_buffer/4_senders/1us_delay execution_time 50.975ms 51.235ms ± 0.115ms 51.225ms ± 0.066ms 51.298ms 51.441ms 51.579ms 51.647ms 0.82% 0.720 0.974 0.22% 0.008ms 1 200
trace_buffer/4_senders/1us_delay throughput 69703.759op/s 70264.235op/s ± 157.468op/s 70277.654op/s ± 90.168op/s 70363.833op/s 70497.805op/s 70581.865op/s 70622.961op/s 0.49% -0.704 0.940 0.22% 11.135op/s 1 200
trace_buffer/4_senders/no_delay execution_time 3.503ms 3.571ms ± 0.029ms 3.571ms ± 0.020ms 3.588ms 3.621ms 3.639ms 3.649ms 2.20% 0.225 -0.195 0.81% 0.002ms 1 200
trace_buffer/4_senders/no_delay throughput 986493.804op/s 1008229.260op/s ± 8129.592op/s 1008168.123op/s ± 5660.417op/s 1014380.921op/s 1019698.399op/s 1025342.034op/s 1027775.637op/s 1.94% -0.183 -0.220 0.80% 574.849op/s 1 200
trace_buffer/8_senders/10us_delay execution_time 60.254ms 60.329ms ± 0.061ms 60.315ms ± 0.030ms 60.343ms 60.459ms 60.534ms 60.578ms 0.44% 1.610 2.590 0.10% 0.004ms 1 200
trace_buffer/8_senders/10us_delay throughput 118854.834op/s 119345.940op/s ± 121.167op/s 119372.936op/s ± 59.897op/s 119433.673op/s 119471.010op/s 119488.551op/s 119493.658op/s 0.10% -1.604 2.564 0.10% 8.568op/s 1 200
trace_buffer/8_senders/1us_delay execution_time 51.355ms 51.817ms ± 0.309ms 51.729ms ± 0.179ms 52.040ms 52.392ms 52.550ms 52.747ms 1.97% 0.821 -0.289 0.59% 0.022ms 1 200
trace_buffer/8_senders/1us_delay throughput 136500.212op/s 138954.306op/s ± 823.844op/s 139185.900op/s ± 481.398op/s 139605.657op/s 139930.244op/s 140077.718op/s 140201.810op/s 0.73% -0.802 -0.326 0.59% 58.255op/s 1 200
trace_buffer/8_senders/no_delay execution_time 7.049ms 7.185ms ± 0.053ms 7.183ms ± 0.036ms 7.218ms 7.271ms 7.326ms 7.343ms 2.22% 0.345 0.085 0.74% 0.004ms 1 200
trace_buffer/8_senders/no_delay throughput 980590.304op/s 1002141.606op/s ± 7387.656op/s 1002378.212op/s ± 5001.772op/s 1007406.314op/s 1012783.142op/s 1017238.179op/s 1021372.503op/s 1.89% -0.301 0.047 0.74% 522.386op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
trace_buffer/1_senders/10us_delay execution_time [60.081ms; 60.091ms] or [-0.009%; +0.009%] None None None
trace_buffer/1_senders/10us_delay throughput [14977.182op/s; 14979.900op/s] or [-0.009%; +0.009%] None None None
trace_buffer/1_senders/1us_delay execution_time [50.338ms; 50.387ms] or [-0.048%; +0.048%] None None None
trace_buffer/1_senders/1us_delay throughput [17862.090op/s; 17879.276op/s] or [-0.048%; +0.048%] None None None
trace_buffer/1_senders/no_delay execution_time [356.340µs; 358.257µs] or [-0.268%; +0.268%] None None None
trace_buffer/1_senders/no_delay throughput [2513761.434op/s; 2525692.598op/s] or [-0.237%; +0.237%] None None None
trace_buffer/2_senders/10us_delay execution_time [60.120ms; 60.132ms] or [-0.010%; +0.010%] None None None
trace_buffer/2_senders/10us_delay throughput [29934.255op/s; 29940.007op/s] or [-0.010%; +0.010%] None None None
trace_buffer/2_senders/1us_delay execution_time [51.055ms; 51.092ms] or [-0.037%; +0.037%] None None None
trace_buffer/2_senders/1us_delay throughput [35230.714op/s; 35256.652op/s] or [-0.037%; +0.037%] None None None
trace_buffer/2_senders/no_delay execution_time [1.669ms; 1.680ms] or [-0.313%; +0.313%] None None None
trace_buffer/2_senders/no_delay throughput [1072209.222op/s; 1078823.273op/s] or [-0.307%; +0.307%] None None None
trace_buffer/4_senders/10us_delay execution_time [60.186ms; 60.198ms] or [-0.009%; +0.009%] None None None
trace_buffer/4_senders/10us_delay throughput [59802.923op/s; 59814.128op/s] or [-0.009%; +0.009%] None None None
trace_buffer/4_senders/1us_delay execution_time [51.219ms; 51.251ms] or [-0.031%; +0.031%] None None None
trace_buffer/4_senders/1us_delay throughput [70242.412op/s; 70286.059op/s] or [-0.031%; +0.031%] None None None
trace_buffer/4_senders/no_delay execution_time [3.567ms; 3.575ms] or [-0.112%; +0.112%] None None None
trace_buffer/4_senders/no_delay throughput [1007102.577op/s; 1009355.943op/s] or [-0.112%; +0.112%] None None None
trace_buffer/8_senders/10us_delay execution_time [60.320ms; 60.337ms] or [-0.014%; +0.014%] None None None
trace_buffer/8_senders/10us_delay throughput [119329.147op/s; 119362.732op/s] or [-0.014%; +0.014%] None None None
trace_buffer/8_senders/1us_delay execution_time [51.775ms; 51.860ms] or [-0.083%; +0.083%] None None None
trace_buffer/8_senders/1us_delay throughput [138840.129op/s; 139068.483op/s] or [-0.082%; +0.082%] None None None
trace_buffer/8_senders/no_delay execution_time [7.178ms; 7.192ms] or [-0.102%; +0.102%] None None None
trace_buffer/8_senders/no_delay throughput [1001117.748op/s; 1003165.464op/s] or [-0.102%; +0.102%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz 643ea5e 1788965317 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
concentrator/add_spans_to_concentrator execution_time 9.605ms 9.633ms ± 0.018ms 9.629ms ± 0.012ms 9.645ms 9.669ms 9.674ms 9.694ms 0.68% 0.783 0.094 0.18% 0.001ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
concentrator/add_spans_to_concentrator execution_time [9.631ms; 9.636ms] or [-0.026%; +0.026%] None None None

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz 643ea5e 1788965317 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
receiver_entry_point/report/2644 execution_time 6.861ms 6.913ms ± 0.031ms 6.907ms ± 0.015ms 6.925ms 6.977ms 7.003ms 7.068ms 2.34% 1.476 3.357 0.44% 0.002ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
receiver_entry_point/report/2644 execution_time [6.909ms; 6.917ms] or [-0.062%; +0.062%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz 643ea5e 1788965317 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
two way interface execution_time 20.902µs 21.632µs ± 0.365µs 21.604µs ± 0.160µs 21.760µs 22.076µs 22.379µs 25.442µs 17.77% 5.891 57.720 1.68% 0.026µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
two way interface execution_time [21.581µs; 21.682µs] or [-0.234%; +0.234%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz 643ea5e 1788965317 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
write only interface execution_time 2.074µs 2.096µs ± 0.009µs 2.094µs ± 0.005µs 2.101µs 2.114µs 2.117µs 2.118µs 1.14% 0.587 -0.291 0.44% 0.001µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
write only interface execution_time [2.095µs; 2.098µs] or [-0.062%; +0.062%] None None None

@litianningdatadog litianningdatadog changed the title feat(shared-runtime): support discarding worker buffers fix(shared-runtime): prevent stale MicroVM runtime data Sep 8, 2026
@dd-octo-sts

dd-octo-sts Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Artifact Size Benchmark Report

aarch64-alpine-linux-musl
Artifact Baseline Commit Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so 8.45 MB 8.45 MB +0% (+144 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a 91.32 MB 91.57 MB +.27% (+253.37 KB) 🔍
aarch64-unknown-linux-gnu
Artifact Baseline Commit Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a 102.61 MB 102.86 MB +.24% (+260.83 KB) 🔍
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so 11.38 MB 11.39 MB +.06% (+7.61 KB) 🔍
libdatadog-x64-windows
Artifact Baseline Commit Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll 27.32 MB 27.40 MB +.26% (+74.50 KB) 🔍
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib 96.08 KB 96.40 KB +.33% (+330 B) 🔍
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb 185.40 MB 185.93 MB +.28% (+544.00 KB) 🔍
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib 804.37 MB 805.65 MB +.15% (+1.28 MB) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll 9.00 MB 9.04 MB +.39% (+36.50 KB) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib 96.08 KB 96.40 KB +.33% (+330 B) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb 26.20 MB 26.26 MB +.20% (+56.00 KB) 🔍
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib 52.21 MB 52.34 MB +.26% (+140.90 KB) 🔍
libdatadog-x86-windows
Artifact Baseline Commit Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll 23.82 MB 23.89 MB +.27% (+68.00 KB) 🔍
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib 97.58 KB 97.91 KB +.33% (+334 B) 🔍
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb 190.46 MB 190.96 MB +.26% (+512.00 KB) 🔍
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib 785.15 MB 788.82 MB +.46% (+3.66 MB) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll 6.96 MB 6.98 MB +.32% (+23.50 KB) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib 97.58 KB 97.91 KB +.33% (+334 B) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb 28.18 MB 28.25 MB +.24% (+72.00 KB) 🔍
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib 49.70 MB 49.82 MB +.25% (+130.39 KB) 🔍
x86_64-alpine-linux-musl
Artifact Baseline Commit Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a 81.48 MB 81.70 MB +.27% (+227.50 KB) 🔍
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so 9.41 MB 9.43 MB +.20% (+20.12 KB) 🔍
x86_64-unknown-linux-gnu
Artifact Baseline Commit Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a 97.21 MB 97.44 MB +.23% (+230.76 KB) 🔍
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so 11.46 MB 11.49 MB +.28% (+33.95 KB) 🔍

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new timeout-based discard shutdown path can cancel discarding mid-flight while consuming self, potentially leaving workers active and undermining the discard-only invariant.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a “discard-only” lifecycle boundary to shared-runtime workers to prevent buffered stats/metrics/telemetry created under a previous AWS Lambda MicroVM runtime identity from being transmitted after an identity refresh.

Changes:

  • Extend libdd_shared_runtime::Worker and WorkerHandle with a discard() pathway that pauses workers and drops buffered state without performing shutdown flushing.
  • Implement discard behavior in key workers (trace stats exporter, telemetry worker, OTLP stats exporter) and add TraceExporter::shutdown_without_flush(_async).
  • Add targeted tests to ensure discard does not emit buffered data and does not run shutdown logic.
File summaries
File Description
libdd-trace-stats/src/stats_exporter.rs Adds discard() to clear concentrator state without sending.
libdd-telemetry/src/worker/mod.rs Adds discard() to reset buffered telemetry and stop lifecycle scheduling.
libdd-shared-runtime/src/worker.rs Extends Worker trait with a default discard() hook.
libdd-shared-runtime/src/shared_runtime/pausable_worker.rs Adds discard() passthrough for paused workers.
libdd-shared-runtime/src/shared_runtime/mod.rs Adds WorkerHandle::discard() to stop a worker without shutdown flushing.
libdd-shared-runtime/src/shared_runtime/fork_safe.rs Adds a test ensuring discard does not invoke shutdown.
libdd-data-pipeline/src/trace_exporter/mod.rs Adds “shutdown without flush” APIs and worker discard orchestration.
libdd-data-pipeline/src/otlp/metrics.rs Adds discard() to clear OTLP metrics concentrator without exporting.
Review details
  • Files reviewed: 8/8 changed files
  • Comments generated: 4
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread libdd-data-pipeline/src/trace_exporter/mod.rs Outdated
Comment thread libdd-data-pipeline/src/otlp/metrics.rs Outdated
Comment thread libdd-shared-runtime/src/shared_runtime/mod.rs Outdated
Comment thread libdd-trace-stats/src/stats_exporter.rs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Discard failures can violate the identity-boundary guarantee, and the lifecycle is unavailable to FFI consumers.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 8/8 changed files
  • Comments generated: 2
  • Review effort level: Balanced

Comment on lines +422 to +425
while let Some(result) = futures.next().await {
if let Err(e) = result {
error!("Worker failed to discard: {:?}", e);
}

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current implementation propagates discard failures rather than treating them as success. discard_workers waits for every WorkerHandle::discard() future, retains the first error, and returns it to shutdown_without_flush.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want to propagate the errors here, these are mostly not actionnable by the tracer and don't matter much in this case since the data is meant ot be discarded anyway. This method should replicate shutdown_workers error handling i.e. logging and returning Ok. Regarding the sync issue with shutdown_async, if a normal shutdown have been triggered data will be flushed and we can't prevent it, there's no reason for this method to wait for the flush to be over.

Comment thread libdd-data-pipeline/src/trace_exporter/mod.rs Outdated
@litianningdatadog
litianningdatadog force-pushed the tianning.li/buffer-reset-upon-microvm-start branch 2 times, most recently from 4bcc5f1 to 8f25ceb Compare September 8, 2026 03:13
@litianningdatadog
litianningdatadog requested a balanced review from Copilot September 8, 2026 03:14

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Teardown can race with fork preparation, and the new public error variant introduces an undeclared breaking Rust API change.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 12/12 changed files
  • Comments generated: 2
  • Review effort level: Balanced

Comment thread libdd-shared-runtime/src/shared_runtime/mod.rs Outdated
Comment thread libdd-data-pipeline/src/trace_exporter/error.rs Outdated
@litianningdatadog
litianningdatadog force-pushed the tianning.li/buffer-reset-upon-microvm-start branch 2 times, most recently from aecc48d to 793eb36 Compare September 8, 2026 03:55
@litianningdatadog
litianningdatadog requested a balanced review from Copilot September 8, 2026 03:56

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Fork-time failures can strand active workers, while teardown locking also changes normal shutdown concurrency.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 11/11 changed files
  • Comments generated: 3
  • Review effort level: Balanced

Comment thread libdd-data-pipeline/src/trace_exporter/mod.rs Outdated
Comment thread libdd-shared-runtime/src/shared_runtime/mod.rs Outdated
Comment thread libdd-shared-runtime/src/worker.rs Outdated
@litianningdatadog
litianningdatadog force-pushed the tianning.li/buffer-reset-upon-microvm-start branch 3 times, most recently from 8a94de9 to f9eca4c Compare September 8, 2026 05:23
@github-actions github-actions Bot removed the telemetry label Sep 8, 2026
@litianningdatadog
litianningdatadog force-pushed the tianning.li/buffer-reset-upon-microvm-start branch from f9eca4c to 01da867 Compare September 8, 2026 13:53
@litianningdatadog
litianningdatadog force-pushed the tianning.li/buffer-reset-upon-microvm-start branch from 01da867 to 8d07f9e Compare September 8, 2026 14:11
@litianningdatadog litianningdatadog added the aws-microvm Necessary work related to AWS Lambda MicroVM label Sep 8, 2026

@hoolioh hoolioh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have you considered controlling the closing behavior through a variable in the exporter builder?
If the case is that you don't have to use both methods (flus and no-flush) in the same flow, IMO, it would be better to encapsulate the behaviour through the exporter handle. That way we will avoid duplicating the external API which comes with some downsides in terms on maintability and binary size.

@litianningdatadog

Copy link
Copy Markdown
Author

have you considered controlling the closing behavior through a variable in the exporter builder? If the case is that you don't have to use both methods (flus and no-flush) in the same flow, IMO, it would be better to encapsulate the behaviour through the exporter handle. That way we will avoid duplicating the external API which comes with some downsides in terms on maintability and binary size.

@hoolioh Thanks for the comments. My thought is this should stay as an explicit close API because the no-flush path has a different ownership/error contract.

On discard failure, the FFI path keeps the exporter handle alive so the caller can retry. The existing ddog_trace_exporter_free consumes the handle and has no way to report an error. If we move this behind a builder flag, free either has to swallow discard failures or we still need a second fallible close function.

I’m fine sharing more internal plumbing to avoid duplicated shutdown code, but I think keeping public discard path explicit is the better and safer approach.

@VianneyRuhlmann VianneyRuhlmann left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The approach is looking fine. However just a heads up, I'm working on making the runtime_id (and other metadata) mutable in the workers. Would it work for you if you were able to update the runtime_id and reset the worker state (we could expose reset on the shared runtime) ?

Comment on lines -1557 to -1558
let buf = [0u8; 32];
let misaligned = buf.as_ptr().add(1).cast::<i64>();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this related to the PR ?

@litianningdatadog litianningdatadog Sep 8, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was exposed by this PR. The original assumption was invalid where [u8; 32] only has alignment 1, so Rust/Miri may place it at any byte address. If buf.as_ptr() happens to be 7 mod 8, adding 1 produces an address that is aligned for i64, so !misaligned.is_aligned() fails. So this is not a public/API breaking change from the PR. It is a test input construction bug surfaced by PR CI/Miri because this PR caused libdd-data-pipeline-ffi tests to run.

Comment thread libdd-shared-runtime/src/shared_runtime/mod.rs Outdated
Comment thread libdd-shared-runtime/src/worker.rs
@litianningdatadog

litianningdatadog commented Sep 8, 2026

Copy link
Copy Markdown
Author

The approach is looking fine. However just a heads up, I'm working on making the runtime_id (and other metadata) mutable in the workers. Would it work for you if you were able to update the runtime_id and reset the worker state (we could expose reset on the shared runtime) ?

@VianneyRuhlmann Thanks for your input. Here is my thought: mutable runtime_id would help avoid rebuilding workers just to refresh future metadata, but we still prefer a no-flush reset boundary for pre-/run state.

Our baseline is that we must not emit data with a stale runtime_id. Retagging buffered data satisfies that mechanically, but it can still produce misleading data. Anything in the buffer before /run may come from snapshot/build-time state, validation traffic, stale worker schedules, or aggregation windows that no longer line up with the new MicroVM instance. The label would be current, but the data would not really belong to that runtime.

That also creates a duplication problem: if the same snapshot contains buffered stats or telemetry, each restored MicroVM could emit the same buffered data under its own fresh runtime_id.

So I think the reset approach in this PR is still the right first step: pause the worker, clear buffered and aggregate state without flushing, then let future work run under the refreshed identity. If mutable worker metadata becomes available later and gives us cleaner lifecycle handling, we can switch to that API in a future release. But I’d still want the MicroVM refresh path to reset/drop pre-refresh state rather than only retag it.

@litianningdatadog
litianningdatadog force-pushed the tianning.li/buffer-reset-upon-microvm-start branch from 45c647c to d881ac4 Compare September 8, 2026 19:04
@litianningdatadog
litianningdatadog marked this pull request as ready for review September 8, 2026 19:14
@litianningdatadog
litianningdatadog requested a review from a team as a code owner September 8, 2026 19:14
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 8, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-08T19:25:07.043030Z d881ac4 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d881ac40f6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread libdd-shared-runtime/src/shared_runtime/mod.rs Outdated
Comment thread libdd-data-pipeline-ffi/src/trace_exporter.rs Outdated
Comment thread libdd-shared-runtime/src/shared_runtime/mod.rs Outdated
Comment thread libdd-data-pipeline/src/trace_exporter/mod.rs
Comment thread .gitignore Outdated
Comment on lines +172 to +178
select! {
biased;
_ = cloned_discard_token.cancelled() => {
break;
}
_ = worker.run() => {}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Threads are restored when the MicroVM is starting from a snapshot right ? This doesn't guarantee that we wont send traces in the new instance:

  1. the thread could send before you call discard
  2. select! can only abort the task when await is called so if run has been called and it doesn't await before flushing (which is often the case since the main reason we need async is for io calls) the stale data will be flushed.

Would it be an option to pause all the threads in /ready or /validate hook like it's done for forks this way in the new image all threads are stopped and there's no risks of flushing stale data.

@litianningdatadog litianningdatadog Sep 9, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're correct. Removing this race entirely would require pausing workers before the MicroVM snapshot is created when the event /ready is fired, then resuming them only after /run refreshes the identity. We are treating that as follow-up work. The RFC accepts the small gap between process restore and completion of /run, because a global hold would also require decisions around queue growth, lost startup diagnostics, lifecycle duplication, and writer/drop accounting. This PR only adds the no-flush path used during /run to discard inherited exporter state before creating the replacement exporter.

@VianneyRuhlmann VianneyRuhlmann Sep 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I see the point of this change then. Either sending stale data with the wrong runtime_id is acceptable, or it's not. If it's acceptable then you don't need to cancel inflight requests or even prevent the flush on shutdown. If it isn't then you need to stop the threads before the snapshot. Otherwise the unwanted flush will happen and since it depends on the state of the workers before the snapshot, an image encountering it will send stale data on every run.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@litianningdatadog
litianningdatadog force-pushed the tianning.li/buffer-reset-upon-microvm-start branch from 5b90477 to 817aa89 Compare September 9, 2026 16:31
Comment thread .gitignore
tmp
__fuzz__

/graphify-out

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is to ignore the output of graphify. But we can remove it if it is decided to use it as a formal tool

@litianningdatadog litianningdatadog changed the title fix(shared-runtime): prevent stale MicroVM runtime data fix(shared-runtime): discard inherited MicroVM runtime data Sep 9, 2026
Comment on lines 43 to +44
pub fn clear_cache_for_test() {
AGENT_INFO_CACHE.store(None);
clear_cache_for_runtime_identity_refresh();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use the new function in the tests directly

let _ = catch_panic!(handle.shutdown(None), Ok(()));
}

/// Discards runtime-identity-bound state without sending buffered data, then frees the exporter.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// Discards runtime-identity-bound state without sending buffered data, then frees the exporter.
/// Stop workers without sending buffered data, then frees the exporter.

It isn't runtime_id specific it deletes all state

runtime.block_on(self.shutdown_async(timeout))?
}

/// Discard inherited runtime-identity state without sending buffered worker data.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// Discard inherited runtime-identity state without sending buffered worker data.
/// Stop workers without sending buffered data.

/// This function is *NOT* cancel safe. If cancelled, workers can be left in an invalid state.
pub async fn shutdown_without_flush_async(self) -> Result<(), TraceExporterError> {
let discard_result = self.discard_workers_without_flush().await;
agent_info::clear_cache_for_runtime_identity_refresh();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this should be called here the method should either be called in the python handler which handles all the snapshot-reset logic or in the agent info fetcher discard.

}
}

async fn discard_workers_without_flush(&self) -> Result<(), TraceExporterError> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To stay coherent with the other method names this should be discard_workers or shutdown_workers_without_flush

Comment on lines +422 to +425
while let Some(result) = futures.next().await {
if let Err(e) = result {
error!("Worker failed to discard: {:?}", e);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want to propagate the errors here, these are mostly not actionnable by the tracer and don't matter much in this case since the data is meant ot be discarded anyway. This method should replicate shutdown_workers error handling i.e. logging and returning Ok. Regarding the sync issue with shutdown_async, if a normal shutdown have been triggered data will be flushed and we can't prevent it, there's no reason for this method to wait for the flush to be over.

/// This function is *NOT* cancel safe and should not be called from
/// [`Worker::trigger`](crate::worker::Worker::trigger). If cancelled, the discarded worker can
/// end up in an invalid state.
pub async fn discard_worker_without_flush(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't be it's own module and should be a method of WorkerHandle also it should be named discard as discard already implies without_flush.

Comment on lines +172 to +178
select! {
biased;
_ = cloned_discard_token.cancelled() => {
break;
}
_ = worker.run() => {}
}

@VianneyRuhlmann VianneyRuhlmann Sep 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I see the point of this change then. Either sending stale data with the wrong runtime_id is acceptable, or it's not. If it's acceptable then you don't need to cancel inflight requests or even prevent the flush on shutdown. If it isn't then you need to stop the threads before the snapshot. Otherwise the unwanted flush will happen and since it depends on the state of the workers before the snapshot, an image encountering it will send stale data on every run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

aws-microvm Necessary work related to AWS Lambda MicroVM data-pipeline

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants