Skip to content

Benchmark: add event-backtest workload to rust_vs_python_benchmark #528

Description

@MDUYN

Context

The examples/rust_vs_python_benchmark/ baseline currently measures only the vector backtest path (app.run_vector_backtests). The Rust reference there represents the best-case ceiling for a future iaf-core Rust kernel.

Most users actually run event backtests (app.run_backtests / app.run_backtest) — the per-bar simulation that mirrors live trading. Its cost profile is meaningfully different from the vector path because the user's run_strategy Python method is invoked on every bar, and the order matcher / portfolio snapshot writer mutate SQLAlchemy objects every tick.

We need a baseline for that workload too, so the iaf-core epic (#521) can track event-backtest improvements separately from vector-backtest improvements.

Proposal

Extend the benchmark with a third workload, event_bench.py, that:

  • Uses the same synthetic OHLCV data generated by generate_data.py
  • Uses the same EMA + RSI strategy logic as python_bench.py (so results are comparable)
  • Calls app.run_backtests(...) (the event-driven path) instead of app.run_vector_backtests(...)
  • Runs the same deterministic parameter grid (same --combos count, same xorshift seed semantics)
  • Writes results/python_event_bench.json with the same schema as python_bench.json

Update compare.py to render a 3-column table:

  • python_event_framework (event-driven, full framework)
  • python_vector_framework (vector, full framework — today's column)
  • rust_reference (ceiling, with SQLite persistence)

Update run_benchmark.sh to run all three.

Why this matters

Event backtests have a harder optimization ceiling than vector backtests because the user's Python run_strategy runs once per bar by design. Estimated landing zones from the iaf-core epic:

Phase Vector ceiling Event ceiling
Today
Phase 1 (#522, #523, #525) — native indicators + streaming + metrics ~5-10× ~5-10×
Phase 2 (#526) — Rust matching engine + bulk SQLite writes ~30-60× ~10-20×
Phase 3 (#521 epic complete) — full vectorized engine / batched callbacks ~80-150× ~30-60×

Without an event-mode baseline we can't validate progress on phases 2 and 3 for the workload that real users care about most.

Acceptance criteria

  • event_bench.py exists and runs to completion against the same data set as python_bench.py
  • results/python_event_bench.json written with implementation: \"python_event_framework\" and matching schema (elapsed_seconds, throughput_backtests_per_second, wall_clock_per_backtest_ms, n_backtests, n_symbols, years, workers, persist)
  • compare.py renders all three columns and prints both speedup ratios (rust vs vector, rust vs event)
  • run_benchmark.sh runs vector + event + rust in sequence, with a SKIP_EVENT=true escape hatch for fast iteration
  • README.md updated to document the event workload and its different ceiling expectations

Related

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performancePerformance / memory optimizationrelease:backlogNot yet targetedrustRust port / native accelerationtype:featureNew capability

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions