fix(shared-runtime): discard inherited MicroVM runtime data - #2478
fix(shared-runtime): discard inherited MicroVM runtime data#2478litianningdatadog wants to merge 3 commits into
Conversation
📚 Documentation Check Results📦
|
🔒 Cargo Deny Results✅ No issues found! 📦
|
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 817aa89 | Docs | View more details | Give us feedback! |
BenchmarksComparisonBenchmark execution time: 2026-09-09 16:43:55 Comparing candidate commit 817aa89 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 25 metrics, 0 unstable metrics.
|
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | 817aa89 | 1788971451 | tianning.li/buffer-reset-upon-microvm-start |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| trace_buffer/1_senders/10us_delay | execution_time | 59.992ms | 60.084ms ± 0.037ms | 60.073ms ± 0.013ms | 60.093ms | 60.156ms | 60.218ms | 60.267ms | 0.32% | 1.862 | 4.928 | 0.06% | 0.003ms | 1 | 200 |
| trace_buffer/1_senders/10us_delay | throughput | 14933.563op/s | 14979.007op/s ± 9.300op/s | 14981.765op/s ± 3.199op/s | 14983.606op/s | 14988.911op/s | 14990.016op/s | 15001.922op/s | 0.13% | -1.856 | 4.901 | 0.06% | 0.658op/s | 1 | 200 |
| trace_buffer/1_senders/1us_delay | execution_time | 50.139ms | 50.366ms ± 0.163ms | 50.332ms ± 0.102ms | 50.453ms | 50.638ms | 50.809ms | 51.207ms | 1.74% | 1.452 | 3.635 | 0.32% | 0.012ms | 1 | 200 |
| trace_buffer/1_senders/1us_delay | throughput | 17575.751op/s | 17869.414op/s ± 57.738op/s | 17881.407op/s ± 36.356op/s | 17909.920op/s | 17942.015op/s | 17948.005op/s | 17950.120op/s | 0.38% | -1.418 | 3.451 | 0.32% | 4.083op/s | 1 | 200 |
| trace_buffer/1_senders/no_delay | execution_time | 351.686µs | 360.000µs ± 7.023µs | 359.327µs ± 1.354µs | 360.724µs | 364.838µs | 376.046µs | 444.608µs | 23.73% | 8.981 | 104.002 | 1.95% | 0.497µs | 1 | 200 |
| trace_buffer/1_senders/no_delay | throughput | 2024253.474op/s | 2500813.220op/s ± 42025.508op/s | 2504680.857op/s ± 9418.390op/s | 2513771.771op/s | 2543880.887op/s | 2554623.973op/s | 2559098.656op/s | 2.17% | -7.584 | 81.517 | 1.68% | 2971.652op/s | 1 | 200 |
| trace_buffer/2_senders/10us_delay | execution_time | 60.071ms | 60.142ms ± 0.052ms | 60.132ms ± 0.031ms | 60.160ms | 60.261ms | 60.282ms | 60.287ms | 0.26% | 1.107 | 0.538 | 0.09% | 0.004ms | 1 | 200 |
| trace_buffer/2_senders/10us_delay | throughput | 29857.225op/s | 29929.081op/s ± 25.944op/s | 29934.245op/s ± 15.208op/s | 29949.526op/s | 29958.403op/s | 29962.105op/s | 29964.355op/s | 0.10% | -1.103 | 0.530 | 0.09% | 1.834op/s | 1 | 200 |
| trace_buffer/2_senders/1us_delay | execution_time | 50.642ms | 51.078ms ± 0.169ms | 51.086ms ± 0.113ms | 51.187ms | 51.351ms | 51.514ms | 51.557ms | 0.92% | 0.174 | -0.017 | 0.33% | 0.012ms | 1 | 200 |
| trace_buffer/2_senders/1us_delay | throughput | 34912.925op/s | 35240.401op/s ± 116.468op/s | 35234.925op/s ± 77.899op/s | 35314.233op/s | 35429.513op/s | 35477.016op/s | 35543.504op/s | 0.88% | -0.155 | -0.031 | 0.33% | 8.236op/s | 1 | 200 |
| trace_buffer/2_senders/no_delay | execution_time | 1.483ms | 1.604ms ± 0.041ms | 1.601ms ± 0.025ms | 1.629ms | 1.682ms | 1.712ms | 1.728ms | 7.94% | 0.243 | 0.314 | 2.57% | 0.003ms | 1 | 200 |
| trace_buffer/2_senders/no_delay | throughput | 1041366.879op/s | 1122851.764op/s ± 28803.856op/s | 1124032.598op/s ± 18089.546op/s | 1141542.551op/s | 1167590.917op/s | 1193891.662op/s | 1213391.652op/s | 7.95% | -0.068 | 0.315 | 2.56% | 2036.740op/s | 1 | 200 |
| trace_buffer/4_senders/10us_delay | execution_time | 60.134ms | 60.186ms ± 0.042ms | 60.173ms ± 0.022ms | 60.202ms | 60.278ms | 60.314ms | 60.385ms | 0.35% | 1.725 | 3.573 | 0.07% | 0.003ms | 1 | 200 |
| trace_buffer/4_senders/10us_delay | throughput | 59617.191op/s | 59814.892op/s ± 41.764op/s | 59827.213op/s ± 21.734op/s | 59843.205op/s | 59858.880op/s | 59865.343op/s | 59866.566op/s | 0.07% | -1.720 | 3.546 | 0.07% | 2.953op/s | 1 | 200 |
| trace_buffer/4_senders/1us_delay | execution_time | 51.081ms | 51.322ms ± 0.158ms | 51.286ms ± 0.071ms | 51.375ms | 51.651ms | 51.923ms | 51.958ms | 1.31% | 1.600 | 3.134 | 0.31% | 0.011ms | 1 | 200 |
| trace_buffer/4_senders/1us_delay | throughput | 69287.073op/s | 70146.039op/s ± 214.314op/s | 70195.184op/s ± 96.580op/s | 70267.364op/s | 70396.607op/s | 70454.893op/s | 70476.460op/s | 0.40% | -1.576 | 3.037 | 0.30% | 15.154op/s | 1 | 200 |
| trace_buffer/4_senders/no_delay | execution_time | 3.476ms | 3.584ms ± 0.046ms | 3.583ms ± 0.023ms | 3.605ms | 3.639ms | 3.665ms | 4.023ms | 12.27% | 4.128 | 38.476 | 1.29% | 0.003ms | 1 | 200 |
| trace_buffer/4_senders/no_delay | throughput | 894822.156op/s | 1004680.646op/s ± 12420.706op/s | 1004628.579op/s ± 6544.462op/s | 1011786.973op/s | 1021321.618op/s | 1029122.847op/s | 1035697.978op/s | 3.09% | -3.302 | 29.106 | 1.23% | 878.277op/s | 1 | 200 |
| trace_buffer/8_senders/10us_delay | execution_time | 60.248ms | 60.336ms ± 0.066ms | 60.319ms ± 0.037ms | 60.378ms | 60.455ms | 60.540ms | 60.651ms | 0.55% | 1.328 | 2.269 | 0.11% | 0.005ms | 1 | 200 |
| trace_buffer/8_senders/10us_delay | throughput | 118712.296op/s | 119331.401op/s ± 130.718op/s | 119365.042op/s ± 73.074op/s | 119432.507op/s | 119471.209op/s | 119498.159op/s | 119505.228op/s | 0.12% | -1.320 | 2.227 | 0.11% | 9.243op/s | 1 | 200 |
| trace_buffer/8_senders/1us_delay | execution_time | 51.251ms | 51.672ms ± 0.320ms | 51.540ms ± 0.143ms | 51.825ms | 52.332ms | 52.480ms | 52.633ms | 2.12% | 1.069 | 0.017 | 0.62% | 0.023ms | 1 | 200 |
| trace_buffer/8_senders/1us_delay | throughput | 136796.223op/s | 139345.558op/s ± 856.360op/s | 139698.363op/s ± 388.013op/s | 139983.038op/s | 140201.853op/s | 140299.648op/s | 140484.639op/s | 0.56% | -1.052 | -0.024 | 0.61% | 60.554op/s | 1 | 200 |
| trace_buffer/8_senders/no_delay | execution_time | 6.976ms | 7.115ms ± 0.082ms | 7.099ms ± 0.055ms | 7.169ms | 7.279ms | 7.326ms | 7.337ms | 3.35% | 0.697 | -0.312 | 1.15% | 0.006ms | 1 | 200 |
| trace_buffer/8_senders/no_delay | throughput | 981348.785op/s | 1012142.549op/s ± 11541.799op/s | 1014216.284op/s ± 7821.056op/s | 1021111.408op/s | 1027303.664op/s | 1029223.962op/s | 1032170.410op/s | 1.77% | -0.656 | -0.382 | 1.14% | 816.128op/s | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| trace_buffer/1_senders/10us_delay | execution_time | [60.079ms; 60.089ms] or [-0.009%; +0.009%] | None | None | None |
| trace_buffer/1_senders/10us_delay | throughput | [14977.719op/s; 14980.296op/s] or [-0.009%; +0.009%] | None | None | None |
| trace_buffer/1_senders/1us_delay | execution_time | [50.343ms; 50.389ms] or [-0.045%; +0.045%] | None | None | None |
| trace_buffer/1_senders/1us_delay | throughput | [17861.412op/s; 17877.416op/s] or [-0.045%; +0.045%] | None | None | None |
| trace_buffer/1_senders/no_delay | execution_time | [359.027µs; 360.973µs] or [-0.270%; +0.270%] | None | None | None |
| trace_buffer/1_senders/no_delay | throughput | [2494988.889op/s; 2506637.551op/s] or [-0.233%; +0.233%] | None | None | None |
| trace_buffer/2_senders/10us_delay | execution_time | [60.135ms; 60.149ms] or [-0.012%; +0.012%] | None | None | None |
| trace_buffer/2_senders/10us_delay | throughput | [29925.485op/s; 29932.676op/s] or [-0.012%; +0.012%] | None | None | None |
| trace_buffer/2_senders/1us_delay | execution_time | [51.055ms; 51.102ms] or [-0.046%; +0.046%] | None | None | None |
| trace_buffer/2_senders/1us_delay | throughput | [35224.260op/s; 35256.542op/s] or [-0.046%; +0.046%] | None | None | None |
| trace_buffer/2_senders/no_delay | execution_time | [1.598ms; 1.610ms] or [-0.357%; +0.357%] | None | None | None |
| trace_buffer/2_senders/no_delay | throughput | [1118859.826op/s; 1126843.701op/s] or [-0.356%; +0.356%] | None | None | None |
| trace_buffer/4_senders/10us_delay | execution_time | [60.180ms; 60.192ms] or [-0.010%; +0.010%] | None | None | None |
| trace_buffer/4_senders/10us_delay | throughput | [59809.104op/s; 59820.680op/s] or [-0.010%; +0.010%] | None | None | None |
| trace_buffer/4_senders/1us_delay | execution_time | [51.300ms; 51.344ms] or [-0.043%; +0.043%] | None | None | None |
| trace_buffer/4_senders/1us_delay | throughput | [70116.337op/s; 70175.741op/s] or [-0.042%; +0.042%] | None | None | None |
| trace_buffer/4_senders/no_delay | execution_time | [3.577ms; 3.590ms] or [-0.180%; +0.180%] | None | None | None |
| trace_buffer/4_senders/no_delay | throughput | [1002959.256op/s; 1006402.037op/s] or [-0.171%; +0.171%] | None | None | None |
| trace_buffer/8_senders/10us_delay | execution_time | [60.327ms; 60.345ms] or [-0.015%; +0.015%] | None | None | None |
| trace_buffer/8_senders/10us_delay | throughput | [119313.285op/s; 119349.517op/s] or [-0.015%; +0.015%] | None | None | None |
| trace_buffer/8_senders/1us_delay | execution_time | [51.628ms; 51.716ms] or [-0.086%; +0.086%] | None | None | None |
| trace_buffer/8_senders/1us_delay | throughput | [139226.875op/s; 139464.242op/s] or [-0.085%; +0.085%] | None | None | None |
| trace_buffer/8_senders/no_delay | execution_time | [7.103ms; 7.126ms] or [-0.159%; +0.159%] | None | None | None |
| trace_buffer/8_senders/no_delay | throughput | [1010542.966op/s; 1013742.131op/s] or [-0.158%; +0.158%] | None | None | None |
Group 2
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | 817aa89 | 1788971451 | tianning.li/buffer-reset-upon-microvm-start |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| concentrator/add_spans_to_concentrator | execution_time | 9.743ms | 9.771ms ± 0.018ms | 9.765ms ± 0.010ms | 9.780ms | 9.807ms | 9.822ms | 9.846ms | 0.83% | 1.241 | 1.701 | 0.19% | 0.001ms | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| concentrator/add_spans_to_concentrator | execution_time | [9.768ms; 9.774ms] or [-0.026%; +0.026%] | None | None | None |
Group 1
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | 817aa89 | 1788971451 | tianning.li/buffer-reset-upon-microvm-start |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| receiver_entry_point/report/2644 | execution_time | 6.861ms | 6.953ms ± 0.027ms | 6.956ms ± 0.016ms | 6.971ms | 6.992ms | 7.011ms | 7.025ms | 1.00% | -0.656 | 1.118 | 0.38% | 0.002ms | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| receiver_entry_point/report/2644 | execution_time | [6.950ms; 6.957ms] or [-0.053%; +0.053%] | None | None | None |
Group 2
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | 817aa89 | 1788971451 | tianning.li/buffer-reset-upon-microvm-start |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| two way interface | execution_time | 20.767µs | 21.324µs ± 0.267µs | 21.296µs ± 0.158µs | 21.470µs | 21.769µs | 22.061µs | 22.647µs | 6.34% | 1.227 | 3.512 | 1.25% | 0.019µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| two way interface | execution_time | [21.287µs; 21.361µs] or [-0.173%; +0.173%] | None | None | None |
Group 3
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | 817aa89 | 1788971451 | tianning.li/buffer-reset-upon-microvm-start |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| write only interface | execution_time | 2.052µs | 2.076µs ± 0.010µs | 2.074µs ± 0.007µs | 2.084µs | 2.093µs | 2.097µs | 2.102µs | 1.33% | 0.351 | -0.545 | 0.46% | 0.001µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| write only interface | execution_time | [2.075µs; 2.078µs] or [-0.063%; +0.063%] | None | None | None |
Baseline
Baseline benchmark details
Group 1
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | 643ea5e | 1788965317 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| trace_buffer/1_senders/10us_delay | execution_time | 59.975ms | 60.086ms ± 0.039ms | 60.076ms ± 0.015ms | 60.100ms | 60.161ms | 60.230ms | 60.263ms | 0.31% | 1.650 | 4.348 | 0.07% | 0.003ms | 1 | 200 |
| trace_buffer/1_senders/10us_delay | throughput | 14934.581op/s | 14978.541op/s ± 9.807op/s | 14980.971op/s ± 3.838op/s | 14983.834op/s | 14988.938op/s | 14991.501op/s | 15006.294op/s | 0.17% | -1.642 | 4.323 | 0.07% | 0.693op/s | 1 | 200 |
| trace_buffer/1_senders/1us_delay | execution_time | 50.094ms | 50.362ms ± 0.176ms | 50.326ms ± 0.079ms | 50.410ms | 50.686ms | 50.993ms | 51.252ms | 1.84% | 1.959 | 5.628 | 0.35% | 0.012ms | 1 | 200 |
| trace_buffer/1_senders/1us_delay | throughput | 17560.406op/s | 17870.683op/s ± 62.001op/s | 17883.410op/s ± 28.032op/s | 17908.190op/s | 17938.710op/s | 17957.601op/s | 17966.096op/s | 0.46% | -1.920 | 5.412 | 0.35% | 4.384op/s | 1 | 200 |
| trace_buffer/1_senders/no_delay | execution_time | 349.807µs | 357.299µs ± 6.918µs | 356.243µs ± 2.530µs | 358.908µs | 365.176µs | 370.583µs | 436.165µs | 22.43% | 7.631 | 82.809 | 1.93% | 0.489µs | 1 | 200 |
| trace_buffer/1_senders/no_delay | throughput | 2063437.054op/s | 2519727.016op/s ± 43044.703op/s | 2526364.600op/s ± 18039.411op/s | 2542454.704op/s | 2559385.470op/s | 2566922.093op/s | 2572844.797op/s | 1.84% | -6.229 | 61.536 | 1.70% | 3043.720op/s | 1 | 200 |
| trace_buffer/2_senders/10us_delay | execution_time | 60.071ms | 60.126ms ± 0.042ms | 60.113ms ± 0.021ms | 60.143ms | 60.224ms | 60.249ms | 60.251ms | 0.23% | 1.327 | 1.154 | 0.07% | 0.003ms | 1 | 200 |
| trace_buffer/2_senders/10us_delay | throughput | 29875.199op/s | 29937.131op/s ± 20.754op/s | 29943.730op/s ± 10.257op/s | 29950.886op/s | 29959.916op/s | 29962.015op/s | 29964.533op/s | 0.07% | -1.324 | 1.146 | 0.07% | 1.467op/s | 1 | 200 |
| trace_buffer/2_senders/1us_delay | execution_time | 50.654ms | 51.073ms ± 0.136ms | 51.083ms ± 0.092ms | 51.169ms | 51.271ms | 51.342ms | 51.466ms | 0.75% | -0.274 | 0.183 | 0.26% | 0.010ms | 1 | 200 |
| trace_buffer/2_senders/1us_delay | throughput | 34974.553op/s | 35243.683op/s ± 93.577op/s | 35236.837op/s ± 63.674op/s | 35305.352op/s | 35392.417op/s | 35475.073op/s | 35535.129op/s | 0.85% | 0.291 | 0.196 | 0.26% | 6.617op/s | 1 | 200 |
| trace_buffer/2_senders/no_delay | execution_time | 1.581ms | 1.674ms ± 0.038ms | 1.673ms ± 0.026ms | 1.699ms | 1.729ms | 1.750ms | 1.898ms | 13.46% | 0.966 | 4.956 | 2.25% | 0.003ms | 1 | 200 |
| trace_buffer/2_senders/no_delay | throughput | 948369.109op/s | 1075516.247op/s ± 23861.867op/s | 1076044.297op/s ± 16500.031op/s | 1092685.725op/s | 1107448.591op/s | 1130602.224op/s | 1138605.429op/s | 5.81% | -0.611 | 3.046 | 2.21% | 1687.289op/s | 1 | 200 |
| trace_buffer/4_senders/10us_delay | execution_time | 60.133ms | 60.192ms ± 0.041ms | 60.180ms ± 0.020ms | 60.209ms | 60.278ms | 60.305ms | 60.315ms | 0.22% | 1.125 | 0.624 | 0.07% | 0.003ms | 1 | 200 |
| trace_buffer/4_senders/10us_delay | throughput | 59686.665op/s | 59808.525op/s ± 40.426op/s | 59820.540op/s ± 20.052op/s | 59836.963op/s | 59855.978op/s | 59860.057op/s | 59867.175op/s | 0.08% | -1.122 | 0.617 | 0.07% | 2.859op/s | 1 | 200 |
| trace_buffer/4_senders/1us_delay | execution_time | 50.975ms | 51.235ms ± 0.115ms | 51.225ms ± 0.066ms | 51.298ms | 51.441ms | 51.579ms | 51.647ms | 0.82% | 0.720 | 0.974 | 0.22% | 0.008ms | 1 | 200 |
| trace_buffer/4_senders/1us_delay | throughput | 69703.759op/s | 70264.235op/s ± 157.468op/s | 70277.654op/s ± 90.168op/s | 70363.833op/s | 70497.805op/s | 70581.865op/s | 70622.961op/s | 0.49% | -0.704 | 0.940 | 0.22% | 11.135op/s | 1 | 200 |
| trace_buffer/4_senders/no_delay | execution_time | 3.503ms | 3.571ms ± 0.029ms | 3.571ms ± 0.020ms | 3.588ms | 3.621ms | 3.639ms | 3.649ms | 2.20% | 0.225 | -0.195 | 0.81% | 0.002ms | 1 | 200 |
| trace_buffer/4_senders/no_delay | throughput | 986493.804op/s | 1008229.260op/s ± 8129.592op/s | 1008168.123op/s ± 5660.417op/s | 1014380.921op/s | 1019698.399op/s | 1025342.034op/s | 1027775.637op/s | 1.94% | -0.183 | -0.220 | 0.80% | 574.849op/s | 1 | 200 |
| trace_buffer/8_senders/10us_delay | execution_time | 60.254ms | 60.329ms ± 0.061ms | 60.315ms ± 0.030ms | 60.343ms | 60.459ms | 60.534ms | 60.578ms | 0.44% | 1.610 | 2.590 | 0.10% | 0.004ms | 1 | 200 |
| trace_buffer/8_senders/10us_delay | throughput | 118854.834op/s | 119345.940op/s ± 121.167op/s | 119372.936op/s ± 59.897op/s | 119433.673op/s | 119471.010op/s | 119488.551op/s | 119493.658op/s | 0.10% | -1.604 | 2.564 | 0.10% | 8.568op/s | 1 | 200 |
| trace_buffer/8_senders/1us_delay | execution_time | 51.355ms | 51.817ms ± 0.309ms | 51.729ms ± 0.179ms | 52.040ms | 52.392ms | 52.550ms | 52.747ms | 1.97% | 0.821 | -0.289 | 0.59% | 0.022ms | 1 | 200 |
| trace_buffer/8_senders/1us_delay | throughput | 136500.212op/s | 138954.306op/s ± 823.844op/s | 139185.900op/s ± 481.398op/s | 139605.657op/s | 139930.244op/s | 140077.718op/s | 140201.810op/s | 0.73% | -0.802 | -0.326 | 0.59% | 58.255op/s | 1 | 200 |
| trace_buffer/8_senders/no_delay | execution_time | 7.049ms | 7.185ms ± 0.053ms | 7.183ms ± 0.036ms | 7.218ms | 7.271ms | 7.326ms | 7.343ms | 2.22% | 0.345 | 0.085 | 0.74% | 0.004ms | 1 | 200 |
| trace_buffer/8_senders/no_delay | throughput | 980590.304op/s | 1002141.606op/s ± 7387.656op/s | 1002378.212op/s ± 5001.772op/s | 1007406.314op/s | 1012783.142op/s | 1017238.179op/s | 1021372.503op/s | 1.89% | -0.301 | 0.047 | 0.74% | 522.386op/s | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| trace_buffer/1_senders/10us_delay | execution_time | [60.081ms; 60.091ms] or [-0.009%; +0.009%] | None | None | None |
| trace_buffer/1_senders/10us_delay | throughput | [14977.182op/s; 14979.900op/s] or [-0.009%; +0.009%] | None | None | None |
| trace_buffer/1_senders/1us_delay | execution_time | [50.338ms; 50.387ms] or [-0.048%; +0.048%] | None | None | None |
| trace_buffer/1_senders/1us_delay | throughput | [17862.090op/s; 17879.276op/s] or [-0.048%; +0.048%] | None | None | None |
| trace_buffer/1_senders/no_delay | execution_time | [356.340µs; 358.257µs] or [-0.268%; +0.268%] | None | None | None |
| trace_buffer/1_senders/no_delay | throughput | [2513761.434op/s; 2525692.598op/s] or [-0.237%; +0.237%] | None | None | None |
| trace_buffer/2_senders/10us_delay | execution_time | [60.120ms; 60.132ms] or [-0.010%; +0.010%] | None | None | None |
| trace_buffer/2_senders/10us_delay | throughput | [29934.255op/s; 29940.007op/s] or [-0.010%; +0.010%] | None | None | None |
| trace_buffer/2_senders/1us_delay | execution_time | [51.055ms; 51.092ms] or [-0.037%; +0.037%] | None | None | None |
| trace_buffer/2_senders/1us_delay | throughput | [35230.714op/s; 35256.652op/s] or [-0.037%; +0.037%] | None | None | None |
| trace_buffer/2_senders/no_delay | execution_time | [1.669ms; 1.680ms] or [-0.313%; +0.313%] | None | None | None |
| trace_buffer/2_senders/no_delay | throughput | [1072209.222op/s; 1078823.273op/s] or [-0.307%; +0.307%] | None | None | None |
| trace_buffer/4_senders/10us_delay | execution_time | [60.186ms; 60.198ms] or [-0.009%; +0.009%] | None | None | None |
| trace_buffer/4_senders/10us_delay | throughput | [59802.923op/s; 59814.128op/s] or [-0.009%; +0.009%] | None | None | None |
| trace_buffer/4_senders/1us_delay | execution_time | [51.219ms; 51.251ms] or [-0.031%; +0.031%] | None | None | None |
| trace_buffer/4_senders/1us_delay | throughput | [70242.412op/s; 70286.059op/s] or [-0.031%; +0.031%] | None | None | None |
| trace_buffer/4_senders/no_delay | execution_time | [3.567ms; 3.575ms] or [-0.112%; +0.112%] | None | None | None |
| trace_buffer/4_senders/no_delay | throughput | [1007102.577op/s; 1009355.943op/s] or [-0.112%; +0.112%] | None | None | None |
| trace_buffer/8_senders/10us_delay | execution_time | [60.320ms; 60.337ms] or [-0.014%; +0.014%] | None | None | None |
| trace_buffer/8_senders/10us_delay | throughput | [119329.147op/s; 119362.732op/s] or [-0.014%; +0.014%] | None | None | None |
| trace_buffer/8_senders/1us_delay | execution_time | [51.775ms; 51.860ms] or [-0.083%; +0.083%] | None | None | None |
| trace_buffer/8_senders/1us_delay | throughput | [138840.129op/s; 139068.483op/s] or [-0.082%; +0.082%] | None | None | None |
| trace_buffer/8_senders/no_delay | execution_time | [7.178ms; 7.192ms] or [-0.102%; +0.102%] | None | None | None |
| trace_buffer/8_senders/no_delay | throughput | [1001117.748op/s; 1003165.464op/s] or [-0.102%; +0.102%] | None | None | None |
Group 2
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | 643ea5e | 1788965317 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| concentrator/add_spans_to_concentrator | execution_time | 9.605ms | 9.633ms ± 0.018ms | 9.629ms ± 0.012ms | 9.645ms | 9.669ms | 9.674ms | 9.694ms | 0.68% | 0.783 | 0.094 | 0.18% | 0.001ms | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| concentrator/add_spans_to_concentrator | execution_time | [9.631ms; 9.636ms] or [-0.026%; +0.026%] | None | None | None |
Group 1
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | 643ea5e | 1788965317 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| receiver_entry_point/report/2644 | execution_time | 6.861ms | 6.913ms ± 0.031ms | 6.907ms ± 0.015ms | 6.925ms | 6.977ms | 7.003ms | 7.068ms | 2.34% | 1.476 | 3.357 | 0.44% | 0.002ms | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| receiver_entry_point/report/2644 | execution_time | [6.909ms; 6.917ms] or [-0.062%; +0.062%] | None | None | None |
Group 2
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | 643ea5e | 1788965317 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| two way interface | execution_time | 20.902µs | 21.632µs ± 0.365µs | 21.604µs ± 0.160µs | 21.760µs | 22.076µs | 22.379µs | 25.442µs | 17.77% | 5.891 | 57.720 | 1.68% | 0.026µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| two way interface | execution_time | [21.581µs; 21.682µs] or [-0.234%; +0.234%] | None | None | None |
Group 3
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | 643ea5e | 1788965317 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| write only interface | execution_time | 2.074µs | 2.096µs ± 0.009µs | 2.094µs ± 0.005µs | 2.101µs | 2.114µs | 2.117µs | 2.118µs | 1.14% | 0.587 | -0.291 | 0.44% | 0.001µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| write only interface | execution_time | [2.095µs; 2.098µs] or [-0.062%; +0.062%] | None | None | None |
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
There was a problem hiding this comment.
🟡 Changes recommended
The new timeout-based discard shutdown path can cancel discarding mid-flight while consuming self, potentially leaving workers active and undermining the discard-only invariant.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds a “discard-only” lifecycle boundary to shared-runtime workers to prevent buffered stats/metrics/telemetry created under a previous AWS Lambda MicroVM runtime identity from being transmitted after an identity refresh.
Changes:
- Extend
libdd_shared_runtime::WorkerandWorkerHandlewith adiscard()pathway that pauses workers and drops buffered state without performing shutdown flushing. - Implement discard behavior in key workers (trace stats exporter, telemetry worker, OTLP stats exporter) and add
TraceExporter::shutdown_without_flush(_async). - Add targeted tests to ensure discard does not emit buffered data and does not run shutdown logic.
File summaries
| File | Description |
|---|---|
| libdd-trace-stats/src/stats_exporter.rs | Adds discard() to clear concentrator state without sending. |
| libdd-telemetry/src/worker/mod.rs | Adds discard() to reset buffered telemetry and stop lifecycle scheduling. |
| libdd-shared-runtime/src/worker.rs | Extends Worker trait with a default discard() hook. |
| libdd-shared-runtime/src/shared_runtime/pausable_worker.rs | Adds discard() passthrough for paused workers. |
| libdd-shared-runtime/src/shared_runtime/mod.rs | Adds WorkerHandle::discard() to stop a worker without shutdown flushing. |
| libdd-shared-runtime/src/shared_runtime/fork_safe.rs | Adds a test ensuring discard does not invoke shutdown. |
| libdd-data-pipeline/src/trace_exporter/mod.rs | Adds “shutdown without flush” APIs and worker discard orchestration. |
| libdd-data-pipeline/src/otlp/metrics.rs | Adds discard() to clear OTLP metrics concentrator without exporting. |
Review details
- Files reviewed: 8/8 changed files
- Comments generated: 4
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
6159184 to
0f6e221
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Discard failures can violate the identity-boundary guarantee, and the lifecycle is unavailable to FFI consumers.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 8/8 changed files
- Comments generated: 2
- Review effort level: Balanced
| while let Some(result) = futures.next().await { | ||
| if let Err(e) = result { | ||
| error!("Worker failed to discard: {:?}", e); | ||
| } |
There was a problem hiding this comment.
The current implementation propagates discard failures rather than treating them as success. discard_workers waits for every WorkerHandle::discard() future, retains the first error, and returns it to shutdown_without_flush.
There was a problem hiding this comment.
We don't want to propagate the errors here, these are mostly not actionnable by the tracer and don't matter much in this case since the data is meant ot be discarded anyway. This method should replicate shutdown_workers error handling i.e. logging and returning Ok. Regarding the sync issue with shutdown_async, if a normal shutdown have been triggered data will be flushed and we can't prevent it, there's no reason for this method to wait for the flush to be over.
4bcc5f1 to
8f25ceb
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Teardown can race with fork preparation, and the new public error variant introduces an undeclared breaking Rust API change.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 12/12 changed files
- Comments generated: 2
- Review effort level: Balanced
aecc48d to
793eb36
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Fork-time failures can strand active workers, while teardown locking also changes normal shutdown concurrency.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 11/11 changed files
- Comments generated: 3
- Review effort level: Balanced
8a94de9 to
f9eca4c
Compare
f9eca4c to
01da867
Compare
01da867 to
8d07f9e
Compare
hoolioh
left a comment
There was a problem hiding this comment.
have you considered controlling the closing behavior through a variable in the exporter builder?
If the case is that you don't have to use both methods (flus and no-flush) in the same flow, IMO, it would be better to encapsulate the behaviour through the exporter handle. That way we will avoid duplicating the external API which comes with some downsides in terms on maintability and binary size.
@hoolioh Thanks for the comments. My thought is this should stay as an explicit close API because the no-flush path has a different ownership/error contract. On discard failure, the FFI path keeps the exporter handle alive so the caller can retry. The existing I’m fine sharing more internal plumbing to avoid duplicated shutdown code, but I think keeping public discard path explicit is the better and safer approach. |
VianneyRuhlmann
left a comment
There was a problem hiding this comment.
The approach is looking fine. However just a heads up, I'm working on making the runtime_id (and other metadata) mutable in the workers. Would it work for you if you were able to update the runtime_id and reset the worker state (we could expose reset on the shared runtime) ?
| let buf = [0u8; 32]; | ||
| let misaligned = buf.as_ptr().add(1).cast::<i64>(); |
There was a problem hiding this comment.
Is this related to the PR ?
There was a problem hiding this comment.
It was exposed by this PR. The original assumption was invalid where [u8; 32] only has alignment 1, so Rust/Miri may place it at any byte address. If buf.as_ptr() happens to be 7 mod 8, adding 1 produces an address that is aligned for i64, so !misaligned.is_aligned() fails. So this is not a public/API breaking change from the PR. It is a test input construction bug surfaced by PR CI/Miri because this PR caused libdd-data-pipeline-ffi tests to run.
@VianneyRuhlmann Thanks for your input. Here is my thought: mutable Our baseline is that we must not emit data with a stale That also creates a duplication problem: if the same snapshot contains buffered stats or telemetry, each restored MicroVM could emit the same buffered data under its own fresh So I think the reset approach in this PR is still the right first step: pause the worker, clear buffered and aggregate state without flushing, then let future work run under the refreshed identity. If mutable worker metadata becomes available later and gives us cleaner lifecycle handling, we can switch to that API in a future release. But I’d still want the MicroVM refresh path to reset/drop pre-refresh state rather than only retag it. |
45c647c to
d881ac4
Compare
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d881ac40f6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| select! { | ||
| biased; | ||
| _ = cloned_discard_token.cancelled() => { | ||
| break; | ||
| } | ||
| _ = worker.run() => {} | ||
| } |
There was a problem hiding this comment.
Threads are restored when the MicroVM is starting from a snapshot right ? This doesn't guarantee that we wont send traces in the new instance:
- the thread could send before you call discard
select!can only abort the task when await is called so ifrunhas been called and it doesn'tawaitbefore flushing (which is often the case since the main reason we need async is for io calls) the stale data will be flushed.
Would it be an option to pause all the threads in /ready or /validate hook like it's done for forks this way in the new image all threads are stopped and there's no risks of flushing stale data.
There was a problem hiding this comment.
You're correct. Removing this race entirely would require pausing workers before the MicroVM snapshot is created when the event /ready is fired, then resuming them only after /run refreshes the identity. We are treating that as follow-up work. The RFC accepts the small gap between process restore and completion of /run, because a global hold would also require decisions around queue growth, lost startup diagnostics, lifecycle duplication, and writer/drop accounting. This PR only adds the no-flush path used during /run to discard inherited exporter state before creating the replacement exporter.
There was a problem hiding this comment.
I'm not sure I see the point of this change then. Either sending stale data with the wrong runtime_id is acceptable, or it's not. If it's acceptable then you don't need to cancel inflight requests or even prevent the flush on shutdown. If it isn't then you need to stop the threads before the snapshot. Otherwise the unwanted flush will happen and since it depends on the state of the workers before the snapshot, an image encountering it will send stale data on every run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
5b90477 to
817aa89
Compare
| tmp | ||
| __fuzz__ | ||
|
|
||
| /graphify-out |
There was a problem hiding this comment.
it is to ignore the output of graphify. But we can remove it if it is decided to use it as a formal tool
| pub fn clear_cache_for_test() { | ||
| AGENT_INFO_CACHE.store(None); | ||
| clear_cache_for_runtime_identity_refresh(); |
There was a problem hiding this comment.
use the new function in the tests directly
| let _ = catch_panic!(handle.shutdown(None), Ok(())); | ||
| } | ||
|
|
||
| /// Discards runtime-identity-bound state without sending buffered data, then frees the exporter. |
There was a problem hiding this comment.
| /// Discards runtime-identity-bound state without sending buffered data, then frees the exporter. | |
| /// Stop workers without sending buffered data, then frees the exporter. |
It isn't runtime_id specific it deletes all state
| runtime.block_on(self.shutdown_async(timeout))? | ||
| } | ||
|
|
||
| /// Discard inherited runtime-identity state without sending buffered worker data. |
There was a problem hiding this comment.
| /// Discard inherited runtime-identity state without sending buffered worker data. | |
| /// Stop workers without sending buffered data. |
| /// This function is *NOT* cancel safe. If cancelled, workers can be left in an invalid state. | ||
| pub async fn shutdown_without_flush_async(self) -> Result<(), TraceExporterError> { | ||
| let discard_result = self.discard_workers_without_flush().await; | ||
| agent_info::clear_cache_for_runtime_identity_refresh(); |
There was a problem hiding this comment.
I don't think this should be called here the method should either be called in the python handler which handles all the snapshot-reset logic or in the agent info fetcher discard.
| } | ||
| } | ||
|
|
||
| async fn discard_workers_without_flush(&self) -> Result<(), TraceExporterError> { |
There was a problem hiding this comment.
To stay coherent with the other method names this should be discard_workers or shutdown_workers_without_flush
| while let Some(result) = futures.next().await { | ||
| if let Err(e) = result { | ||
| error!("Worker failed to discard: {:?}", e); | ||
| } |
There was a problem hiding this comment.
We don't want to propagate the errors here, these are mostly not actionnable by the tracer and don't matter much in this case since the data is meant ot be discarded anyway. This method should replicate shutdown_workers error handling i.e. logging and returning Ok. Regarding the sync issue with shutdown_async, if a normal shutdown have been triggered data will be flushed and we can't prevent it, there's no reason for this method to wait for the flush to be over.
| /// This function is *NOT* cancel safe and should not be called from | ||
| /// [`Worker::trigger`](crate::worker::Worker::trigger). If cancelled, the discarded worker can | ||
| /// end up in an invalid state. | ||
| pub async fn discard_worker_without_flush( |
There was a problem hiding this comment.
This shouldn't be it's own module and should be a method of WorkerHandle also it should be named discard as discard already implies without_flush.
| select! { | ||
| biased; | ||
| _ = cloned_discard_token.cancelled() => { | ||
| break; | ||
| } | ||
| _ = worker.run() => {} | ||
| } |
There was a problem hiding this comment.
I'm not sure I see the point of this change then. Either sending stale data with the wrong runtime_id is acceptable, or it's not. If it's acceptable then you don't need to cancel inflight requests or even prevent the flush on shutdown. If it isn't then you need to stop the threads before the snapshot. Otherwise the unwanted flush will happen and since it depends on the state of the workers before the snapshot, an image encountering it will send stale data on every run.
Why
AWS Lambda MicroVM snapshots can restore a process with trace, stats, and telemetry data accumulated under a previous runtime identity. This adds a no-flush identity-refresh path so
/runcallers can discard that inherited exporter state before creating a replacement exporter.It does not eliminate the small window between process restore and completion of the
/runrefresh. Data emitted in that window may still carry the snapshotted runtime ID. The RFC accepts that MVP gap; closing it requires a pre-snapshot worker hold and the associated lifecycle and accounting changes.How
Adds a discard-only exporter teardown for the MicroVM identity-refresh path.
TraceExporter::shutdown_without_flush()races each worker's in-flight async run against cancellation rather than waiting for normal pause/shutdown behavior. It does not preempt synchronous work already running inside a worker or remove the pre-/runrace. It removes the worker throughWorker::discard().discard()defaults toreset(), but lets a worker override cleanup that must differ from its fork-continuationreset(); the trace-buffer worker uses this to close its channel instead of reopening it. That worker is not wired into an exporter yet, but the generic discard contract needs to handle it correctly before it is. The exporter also clears the global agent-info cache so the replacement exporter does not skip its first/infofetch. The FFI exposes this asddog_trace_exporter_free_without_flush.This is intentionally scoped to the identity-refresh API. Existing
WorkerHandle::stop, runtime shutdown, and fork lifecycle behavior are unchanged:pause()still waits for an in-flight run to finish, andreset()still reopens a worker for continued use. Callers use discard in the MicroVM startup identity-refresh path, not concurrently with ordinary shutdown or a real fork.References
Verification
cargo +nightly-2026-07-26 fmt --all -- --checkcargo check -p libdd-shared-runtime -p libdd-data-pipeline -p libdd-data-pipeline-fficargo test -p libdd-shared-runtimecargo test -p libdd-data-pipeline test_shutdown_without_flushcargo test -p libdd-data-pipeline test_worker_discardcargo test -p libdd-data-pipeline-ffi exporter_free_without_flushcargo +stable clippy -p libdd-shared-runtime --all-targets -- -D warnings