Skip to content

chore: derive cassette suites from the tree and sort-check manifest lists - #2437

Open
gold-silver-copper wants to merge 1 commit into
mainfrom
chore/kill-append-only-registries
Open

chore: derive cassette suites from the tree and sort-check manifest lists#2437
gold-silver-copper wants to merge 1 commit into
mainfrom
chore/kill-append-only-registries

Conversation

@gold-silver-copper

Copy link
Copy Markdown
Contributor

Description

Two kinds of hand-maintained list made unrelated PRs collide on the same hunk: the PROVIDER_CASSETTE_SUITES table in tests/common/cassette_safety.rs (one entry per provider, every wrapper name spelled out, 28 entries) and the per-crate lists in Cargo.toml, rig-core's providers module list and the README integration table. This PR removes the first and sort-checks the rest.

Cassette suites are discovered from the tree. A provider is a tests/providers/<provider>/ directory with a tests/<provider>.rs binary; a cassette wrapper is any call to a function named with_*cassette* inside that directory, first argument the scenario. Discovery is call-site based because some providers define wrappers through macros. The table and the wrapper_names field are gone; the module doc states the convention as the invariant.

Every existing failure mode is kept: secrets scan per provider directory, loose files under tests/cassettes/, a cassette directory no binary scans, a non-identifier provider name, a binary that omits the safety module, missing and orphaned cassettes. Two are new:

  • a wrapper naming another provider (with_openai_cassette) called from a different provider's directory fails with an explicit message instead of a confusing missing/orphaned pair;
  • a tests/providers/<x>/ directory with no tests/<x>.rs binary that calls cassette wrappers fails, since nothing would check its scenarios.

The table was also unsound: a wrapper defined but not listed was silently ignored. with_openai_boxed_cassette and with_anthropic_boxed_cassette were in that state; both are now discovered (their scenarios already had cassettes, so nothing became missing or orphaned).

cargo xtask check-sorted-blocks verifies that every list between sorted: start / sorted: end markers is in case-insensitive byte order (a textual check, since line order is the point). Marked: the companion-crate block and features block in Cargo.toml, [dev-dependencies], both parts of [workspace.dependencies], crates/rig-core/src/providers/mod.rs, and the README integration table. It runs in the stable / fmt job. AGENTS.md and CONTRIBUTING.md state the rule.

Decisions:

  • lancedb in [dependencies] is not a companion crate, so it sits just below the sorted rig-* block with a comment, next to the lancedb feature that needs it. rig-derive moved up with the other core deps for the same reason.
  • Sorting is case-insensitive so the README reads ScyllaDB, SQLite rather than SQLite, ScyllaDB.
  • Providers with no cassette directory (azure, huggingface, hyperbolic, minimax, mira, moonshot, together, voyageai, xiaomimimo, zai) are not required to compile the safety module, since common/cassettes.rs is 3,900 lines they have no use for. The moment tests/cassettes/<provider> appears, every binary fails until theirs includes it, which is the same guarantee the table gave.

Changelog

  • (tests, xtask) cassette suites are discovered from the tree and manifest lists are sort-checked; no user-visible change

Migration

None.

Type of change

  • Documentation update

Testing

  • cargo nextest run --features bedrock -p rig -E 'test(cassette_safety)': 36 tests across 18 provider binaries pass.
  • Discovery first surfaced 10 binaries without the safety module (the providers with no cassette directory, above) and 42 forwarding calls in support.rs files where a derived wrapper passes its scenario variable to the base wrapper. The first is handled as described; the second by not treating a wrapper's own body as a call site.
  • Negative probe: a file in tests/providers/anthropic/cassette/ calling with_openai_cassette("agent/completion_smoke", …) and with_anthropic_cassette("does/not/exist", …) fails with the foreign-provider message and a missing cassette; the probe file was removed.
  • cargo test -p xtask sorted_blocks: 6 unit tests (multi-line entries, table header, unclosed block, duplicates, Rust and Markdown entries).
  • cargo xtask check-sorted-blocks: 7 blocks in 3 files ok. check-test-layout and generate-provider-aliases --check ok.
  • cargo fmt -- --check, cargo clippy -p xtask --all-targets, cargo clippy -p rig --tests --features bedrock clean; git diff --stat -- Cargo.lock tests/cassettes empty; .github/scripts/check-frozen-release-docs.sh origin/main passes.

Checklist:

  • My code follows the style guidelines of this project
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I did not edit CHANGELOG.md or MIGRATING.md (they are generated at release)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant