Skip to content

ci: run Strix behavioral evals through OrchestrAI - #244

Closed
saman-amd wants to merge 6 commits into
amd:mainfrom
saman-amd:orchestrai-evals
Closed

saman-amd wants to merge 6 commits into
amd:mainfrom
saman-amd:orchestrai-evals

Conversation

@saman-amd

@saman-amd saman-amd commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Run the existing Linux and Windows Strix Halo behavioral evals through OrchestrAI while preserving the workflow name, required evals / results check, per-skill verdicts, and existing Instinct path.

Details

  • Group selected Strix cases into one Linux and one Windows OrchestrAI run.
  • Test the immutable PR head commit and require enable_orchestrai_ci before using hardware.
  • Use the Playbooks-compatible ORCHESTRAI_CONTROL_RUNNER and ORCHESTRAI_WAIT_RUNNER variables, with ubuntu-latest as the fallback.
  • Keep routing on ubuntu-latest; unrelated label events are no-ops.
  • Add bounded retries, cancellation handling, sanitized verdicts, and ReportPortal View Results links.
  • Support live and allocation-free mock runs through manual dispatch.
  • Add the controller, configuration, reporting helpers, tests, and documentation without changing skills, eval datasets, test-library code, or shared pipeline code.

Validation

  • 54 controller, plan, reporting, and workflow-contract tests pass.
  • Python compilation and diff checks pass.
  • Trusted Linux and Windows live validation completed, and all 10 per-test artifacts included a ReportPortal link.

Authenticated live checks require a maintainer-owned branch because GitHub does not provide target-repository secrets to fork pull requests.

@saman-amd
saman-amd force-pushed the orchestrai-evals branch 2 times, most recently from 8792781 to d082780 Compare September 28, 2026 17:55
@saman-amd
saman-amd marked this pull request as ready for review September 28, 2026 18:07
@danielholanda

Copy link
Copy Markdown
Collaborator

Hi @saman-amd, can you please create this PR from a branch instead of a fork, so the workflows can pick up the right credentials?

@saman-amd

Copy link
Copy Markdown
Collaborator Author

Superseded by #247, which uses a trusted branch in amd/skills so Actions can receive repository credentials.

@saman-amd saman-amd closed this Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants