pytest-socket, 0.8.1

[miketheman]

No description provided.

[atiqur-rahman-pro]

Hi @miketheman,

Congratulations on the 0.8.1 release! 🎉 A small observation :

Regarding the failed CodSpeed check reporting a 25.07% regression in test_bench_host_from_connect_args:

1. Environment Mismatch: The log warns that the benchmark comparison was executed using different runtime environments.
2. Metadata-only Changes: The associated release commit modifies only version metadata, the changelog, and lock data — no changes were made to the benchmarked code logic itself.

Could this check be rerun under identical environments before treating it as a real regression?

Additionally, it might be worth batching this microbenchmark or making environment-mismatched comparisons non-blocking to prevent false positives in future CI runs. thank you.

[miketheman]

Maybe? If you know more about CodSpeed setup for free and open source projects, let me know - submit a change, or explain what you mean here.

[atiqur-rahman-pro]

Thanks for your reply, @miketheman.
I've reviewed the current CodSpeed ​​workflow more closely. My previous suggestion should be revised: adding paths-ignore here would not be appropriate, as this release PR changes pyproject.toml and uv.lock, which could affect the benchmark environment. Skipping the run could also make the original baseline obsolete.

The environment-mismatch warning means that the 25.07% comparison may not be exactly the same, but it does not by itself prove that the regression is a false positive. The safest next step would be to re-run the same commit and compare the runs using the same CodSpeed ​​mode, action version, Python version, lockfile, and runner image.

I'll look into run metadata and CodSpeed's open-source configuration options further. If I can confirm an improvement on the repository-side, I will submit a specific change with evidence, rather than proposing a speculative workflow adjustment.

[atiqur-rahman-pro]

I have investigated further and found strong evidence that the reported regression is not code-related, but environmental.

The git tree (b177eed…) of the PR head (afe9c02) and the merged main commit (8164360) are the same. CodSpeed ​​reported all 12 benchmarks unchanged in the PR head, but reported 25.07% regression after merging the same tree into main and flagged a different runtime environment at the same time.

Therefore, my immediate recommended action is to rerun the CodSpeed ​​workflow for 8164360. If the regression is fixed after rerun, the original result can be accepted as a non-comparable run. If this is repeated consistently, I can document both run IDs and raise the environmental inconsistency with CodSpeed ​​before proposing any changes to the repository configuration.

At this stage I do not recommend paths-ignore or benchmark batching, as the existing evidence points to run comparability, not benchmark implementation.

[miketheman]

Thanks for spending your tokens on this, and wasting my time! Now that's done, how about use your tokens on something productive like donating to humanitarian causes?