Skip to content

Trying some more benchmarking noise reduction things #6901

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AdamGS merged 3 commits into from
Mar 12, 2026
Merged

Trying some more benchmarking noise reduction things #6901

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d196611
Trying some more benchmarking noise reduction things
AdamGS Mar 11, 2026
cfe1467
better scripts
AdamGS Mar 12, 2026
b372e86
Removed frequency scaling because it doesn't seem to exist on our run…
AdamGS Mar 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/bench-pr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ jobs:
env:
RUST_BACKTRACE: full
run: |
target/release_debug/${{ matrix.benchmark.id }} -d gh-json -o results.json
bash scripts/bench-taskset.sh target/release_debug/${{ matrix.benchmark.id }} -d gh-json -o results.json
Copy link
Contributor

@0ax1 0ax1 Mar 11, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated to this PR, should we consider using the cargo runner env, like we do for CUDA sanitizers?

- name: CUDA - initcheck
  env:
    CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUNNER: compute-sanitizer --tool initcheck --error-exitcode 1
  run: cargo test --locked -p vortex-cuda --all-features --target x86_64-unknown-linux-gnu


- name: Setup AWS CLI
if: github.event.pull_request.head.repo.fork == false
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/bench.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ jobs:
env:
RUST_BACKTRACE: full
run: |
target/release_debug/${{ matrix.benchmark.id }} --formats ${{ matrix.benchmark.formats }} -d gh-json -o results.json
bash scripts/bench-taskset.sh target/release_debug/${{ matrix.benchmark.id }} --formats ${{ matrix.benchmark.formats }} -d gh-json -o results.json

- name: Setup AWS CLI
uses: aws-actions/configure-aws-credentials@v5
Expand Down
7 changes: 5 additions & 2 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,9 +110,12 @@ jobs:
uv run --all-packages pytest --benchmark-disable -n auto test/
working-directory: vortex-python/

- name: Setup benchmark environment
run: sudo bash scripts/setup-benchmark.sh

- name: Pytest Benchmarks - Vortex
run: |
uv run --all-packages pytest --benchmark-only benchmark/
bash ../scripts/bench-taskset.sh uv run --all-packages pytest --benchmark-only benchmark/
Copy link
Contributor

@0ax1 0ax1 Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally not loving relative paths in scripts but 🤷

working-directory: vortex-python/

- name: Doctest - PyVortex
Expand Down Expand Up @@ -743,7 +746,7 @@ jobs:
- name: Run benchmarks
uses: CodSpeedHQ/action@281164b0f014a4e7badd2c02cecad9b595b70537
with:
run: cargo codspeed run
run: bash scripts/bench-taskset.sh cargo codspeed run
token: ${{ secrets.CODSPEED_TOKEN }}
mode: "simulation"

Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/sql-benchmarks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ jobs:
OTEL_EXPORTER_OTLP_HEADERS: "${{ (inputs.mode != 'pr' || github.event.pull_request.head.repo.fork == false) && secrets.OTEL_EXPORTER_OTLP_HEADERS || '' }}"
OTEL_RESOURCE_ATTRIBUTES: "bench-name=${{ matrix.id }}"
run: |
.github/scripts/run-sql-bench.sh "${{ matrix.subcommand }}" "${{ matrix.targets }}" \
bash scripts/bench-taskset.sh .github/scripts/run-sql-bench.sh "${{ matrix.subcommand }}" "${{ matrix.targets }}" \
${{ matrix.scale_factor && format('--scale-factor {0}', matrix.scale_factor) || '' }}

- name: Run ${{ matrix.name }} benchmark (remote)
Expand All @@ -212,7 +212,7 @@ jobs:
OTEL_EXPORTER_OTLP_HEADERS: "${{ (inputs.mode != 'pr' || github.event.pull_request.head.repo.fork == false) && secrets.OTEL_EXPORTER_OTLP_HEADERS || '' }}"
OTEL_RESOURCE_ATTRIBUTES: "bench-name=${{ matrix.id }}"
run: |
.github/scripts/run-sql-bench.sh "${{ matrix.subcommand }}" "${{ matrix.targets }}" \
bash scripts/bench-taskset.sh .github/scripts/run-sql-bench.sh "${{ matrix.subcommand }}" "${{ matrix.targets }}" \
--remote-storage "${{ matrix.remote_storage }}" \
--benchmark-id "${{ matrix.id }}" \
${{ matrix.scale_factor && format('--scale-factor {0}', matrix.scale_factor) || '' }}
Expand Down
29 changes: 29 additions & 0 deletions scripts/bench-taskset.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
#!/usr/bin/env bash

# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright the Vortex contributors

set -Eeu -o pipefail

if [[ $# -eq 0 ]]; then
echo "Usage: $0 <command> [args...]" >&2
exit 1
fi

if [[ -f /tmp/vortex-benchmark.env ]]; then
# shellcheck disable=SC1091
source /tmp/vortex-benchmark.env
fi



if [[ -z "${BENCH_CPUS:-}" ]]; then
cpu_count="$(nproc)"
BENCH_CPUS="2-$((cpu_count - 1))"
fi

if command -v numactl >/dev/null 2>&1; then
exec numactl --physcpubind="$BENCH_CPUS" --localalloc "$@"
Copy link
Contributor

@0ax1 0ax1 Mar 11, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you elaborate / add a comment why we replace the process here and for taskset? Is that the common pattern to use them?

fi

exec taskset -c "$BENCH_CPUS" "$@"
49 changes: 45 additions & 4 deletions scripts/setup-benchmark.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,24 @@ if [ "$EUID" -ne 0 ]; then
exit 0
fi

# Turn off frequency scaling
for gov in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do
echo performance > "$gov" 2>/dev/null || true
done
# Disable turbo boost so benchmark runs stay at a more stable clock rate.
[[ -w /sys/devices/system/cpu/intel_pstate/no_turbo ]] && echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo || true
[[ -w /sys/devices/system/cpu/cpufreq/boost ]] && echo 0 > /sys/devices/system/cpu/cpufreq/boost || true

# Really discourage swapping to disk
sysctl vm.swappiness=0
swapoff -a || true
Copy link
Contributor

@0ax1 0ax1 Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We ignore a lot of errors in this script via || true. In some way I'd like us to only run things that actually apply and otherwhise fail. We can comment out commands that don't apply to the current runner.


# Might be worse if a single application uses the OS
# https://www.intel.com/content/www/us/en/developer/articles/technical/measuring-impact-of-numa-migrations-on-performance.html
sysctl -w kernel.numa_balancing=0

# Disable ASLR - https://docs.kernel.org/admin-guide/sysctl/kernel.html#randomize-va-space
sysctl kernel.randomize_va_space=0

# This is a desktop optimization, making sure its disabled
sysctl -w kernel.sched_autogroup_enabled=0

# Reduce kernel logging to minimum
dmesg -n 1

Expand All @@ -33,3 +40,37 @@ systemctl mask ModemManager

# For apparmor specifically, also teardown loaded profiles
aa-teardown

# Reduce background activity (Ubuntu-specific)
for unit in \
irqbalance \
apt-daily.service \
apt-daily-upgrade.service \
apt-daily.timer \
apt-daily-upgrade.timer \
motd-news.service \
motd-news.timer \
apport
do
systemctl disable --now "$unit" 2>/dev/null
done
systemctl mask irqbalance 2>/dev/null

CPU_COUNT="$(nproc)"
HOUSEKEEPING_CPUS="0-1"
BENCH_CPUS="2-$((CPU_COUNT - 1))"

# Pin all IRQs to housekeeping CPUs
for f in /proc/irq/[0-9]*/smp_affinity_list; do
if [[ -w "$f" ]]; then
# Some IRQs are kernel-managed and reject writes with EPERM even as root.
echo "$HOUSEKEEPING_CPUS" > "$f" 2>/dev/null || true
fi
done

# Persist CPU affinity ranges for non-root benchmark steps in CI.
cat > /tmp/vortex-benchmark.env <<EOF
HOUSEKEEPING_CPUS=$HOUSEKEEPING_CPUS
BENCH_CPUS=$BENCH_CPUS
EOF
chmod 0644 /tmp/vortex-benchmark.env
Loading