feat(build): Add benchmarks framework #902
Conversation
MichaelYochpaz
force-pushed
the
add-benchmarks-framework
branch
7 times, most recently
from
9376524 to
2d7bb32
Compare
MichaelYochpaz
force-pushed
the
add-benchmarks-framework
branch
8 times, most recently
from
5481b9e to
2bfd5fb
Compare
|
Hey @python-wheel-build/fromager-maintainers, PR is (finally) ready for review :) Stack
Workflows (GitHub Actions)
Initial Benchmarks
Integration benchmarks use a local PyPI fixture: packages are pre-downloaded during setup, then served locally to measure resolution performance without network variability. Note: This PR focuses on the framework, not comprehensive coverage. We'll add more benchmarks in the future, in separate PRs where it's easy to test and review each new benchmark individually. For a more detailed overview, see benchmarks/README.md. It should cover everything, if it does not - let me know and I'll update it. I've tested CodSpeed on my fork to some degree, but walltime (which has a limited amount of free quota for open-source projects) is only available for GitHub projects hosted on orgs, so I couldn't test it. CodSpeed also released a new feature to benchmark and track memory usage last week, which isn't included in this PR, but we can add in the future with benchmarks that are more focused on testing memory usage. |
| - name: Set up Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: "3.11" |
There was a problem hiding this comment.
Please use 3.12 instead of 3.11.
3.12 is the primary version for deployments. 3.11 is on legacy support.
benchmarks/test_integration.py
Outdated
| Downloads the package to a temp directory, then uses the server's | ||
| add_package() method to place it in the correct PEP 503 structure. | ||
| """ | ||
| from packaging.requirements import Requirement |
benchmarks/test_integration.py
Outdated
| # Create required directories | ||
| patches_dir = tmp_path / "patches" | ||
| patches_dir.mkdir(exist_ok=True) | ||
| sdists_repo = tmp_path / "sdists-repo" | ||
| sdists_repo.mkdir(exist_ok=True) | ||
| wheels_repo = tmp_path / "wheels-repo" | ||
| wheels_repo.mkdir(exist_ok=True) | ||
| work_dir = tmp_path / "work-dir" | ||
| work_dir.mkdir(exist_ok=True) |
There was a problem hiding this comment.
WorkContext.setup creates the directories for you. You don't have to create them manually.
MichaelYochpaz
changed the title
Add performance benchmarking infrastructure for fromager including GitHub Actions workflows for nightly and on-demand runs. Signed-off-by: Michael Yochpaz <myochpaz@redhat.com>
MichaelYochpaz
force-pushed
the
add-benchmarks-framework
branch
from
2bfd5fb to
c3266c2
Compare
2 participants
Add performance benchmarking infrastructure for Fromager.
See the newly created README for detailed information.
A continuation of #901
Closes #734