Skip to content

Commit 244e418

Browse files
codexByron
authored andcommitted
ci: compare CLI and GixPython repository benchmarks
Add a separate `Backend benchmark` job using one CPython 3.12 interpreter and two installations, with official `GixPython==0.1.0` only in the native one. Prepare a fixed SHA-1/files checkout of GitPython 3.1.45 with one branch, an untracked file and an ignored directory before timing. Keep local/global Git configuration and optional locks from changing the measured workload. Run the calibrated `pyperf` suite with GixPython first, then CLI on the same runner. Include separate direct repository opening and nested discovery while keeping the operation journey on a retained `Repo`. Validate result digests and workload metadata, publish all means and standard deviations plus `pyperf` significance reporting in the job summary, and retain raw JSON and comparison artifacts even on failure. Timings are observational on shared runners; execution and parity errors fail the job, without a noisy performance threshold. New `MEASUREMENTS` entries automatically participate in this job. The final fixed workload passed all eleven GixPython-first and CLI-second smoke measurements locally with matching result digests. Four comparison regression cases passed both installations. Workflow YAML parsing, Ruff lint/format and `git diff --check` passed. Hosted execution will follow the branch push.
1 parent afea211 commit 244e418

1 file changed

Lines changed: 77 additions & 0 deletions

File tree

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
name: Backend benchmark
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
workflow_dispatch:
8+
9+
permissions:
10+
contents: read
11+
12+
jobs:
13+
compare-backends:
14+
runs-on: ubuntu-latest
15+
timeout-minutes: 20
16+
steps:
17+
- uses: actions/checkout@v7
18+
with:
19+
fetch-depth: 0
20+
persist-credentials: false
21+
- uses: actions/setup-python@v7
22+
with:
23+
python-version: "3.12"
24+
- name: Ensure Git 2.52 or newer
25+
run: |
26+
if ! dpkg --compare-versions "$(git version | awk '{print $3}')" ge 2.52; then
27+
sudo add-apt-repository --yes ppa:git-core/ppa
28+
sudo apt-get update
29+
sudo apt-get install --yes git
30+
fi
31+
- name: Install separate backends with the same interpreter
32+
run: |
33+
python -m venv .bench-cli
34+
python -m venv .bench-gix
35+
.bench-cli/bin/python -m pip install . -r test/performance/benchmark-requirements.txt
36+
.bench-gix/bin/python -m pip install '.[gix]' -r test/performance/benchmark-requirements.txt
37+
- name: Prepare an existing repository outside the timed operations
38+
env:
39+
BENCH_REPO: ${{ runner.temp }}/benchmark-repo
40+
GIT_CONFIG_NOSYSTEM: "1"
41+
GIT_CONFIG_GLOBAL: /dev/null
42+
run: |
43+
git init --object-format=sha1 --ref-format=files "$BENCH_REPO"
44+
git -C "$BENCH_REPO" fetch --no-tags "$GITHUB_WORKSPACE" 6ba2c0a2f9ee7feffd7e079621c4845820180c9a
45+
git -C "$BENCH_REPO" checkout -b benchmark FETCH_HEAD
46+
git -C "$BENCH_REPO" config core.excludesFile /dev/null
47+
git -C "$BENCH_REPO" config core.attributesFile /dev/null
48+
git -C "$BENCH_REPO" config core.fsmonitor false
49+
git -C "$BENCH_REPO" config core.untrackedCache false
50+
git -C "$BENCH_REPO" config gc.auto 0
51+
git -C "$BENCH_REPO" config maintenance.auto false
52+
mkdir "$BENCH_REPO/.benchmark-cache"
53+
touch "$BENCH_REPO/.benchmark-cache/probe" "$BENCH_REPO/benchmark-untracked.txt"
54+
echo '.benchmark-cache/' >> "$BENCH_REPO/.git/info/exclude"
55+
mkdir benchmark-results
56+
- name: Measure GixPython operations, opening and discovery first
57+
run: |
58+
.bench-gix/bin/python -m test.performance.bench_repository \
59+
--repo "$RUNNER_TEMP/benchmark-repo" --expect-backend gix -o benchmark-results/gix.json
60+
- name: Measure CLI operations, opening and discovery
61+
run: |
62+
.bench-cli/bin/python -m test.performance.bench_repository \
63+
--repo "$RUNNER_TEMP/benchmark-repo" --expect-backend cli -o benchmark-results/cli.json
64+
- name: Check parity and publish comparison
65+
run: |
66+
.bench-gix/bin/python -m test.performance.compare_backends \
67+
benchmark-results/cli.json benchmark-results/gix.json > benchmark-results/comparison.md
68+
cat benchmark-results/comparison.md >> "$GITHUB_STEP_SUMMARY"
69+
.bench-gix/bin/python -m pyperf compare_to \
70+
benchmark-results/cli.json benchmark-results/gix.json --table --table-format md \
71+
>> "$GITHUB_STEP_SUMMARY"
72+
- name: Retain raw timings and comparison
73+
if: always()
74+
uses: actions/upload-artifact@v7
75+
with:
76+
name: backend-benchmark
77+
path: benchmark-results/

0 commit comments

Comments
 (0)