AI Test Toolkit: language-first data-driven eval support#9460
Draft
thloke wants to merge 5 commits into
Draft
Conversation
…ce) support Adds a shared ITestDataSource provider, an 'AIT Test Case Context' interface (extends ITestContext) and a delegating context, a per-case dataset-lineage bridge for logging, dual-mode dataset resolution, and coexistence with the legacy per-row expansion (auto-detected via CodeUnit Metadata.HasTestDataSource, with a Language First line flag as override). Docs updated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…n pattern Adds a step-by-step migration recipe (attribute + context-parameter + drop the AIT Test Context var), before/after example, rules (whole-codeunit, auto-detect), variations (multi-turn, harms/adversarial, custom providers) and a checklist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Migrating AIT evals to the language-first [TestDataSource] construct moves the per-case fan-out to the platform test runner, so the toolkit's classic per-case bracketing (Test Runner - Mgt.OnBefore/OnAfterTestMethodRun subscribers, which only run under an AIT suite) no longer engages. This adds the platform-native seam: - codeunit 149050 "AIT Test Handler" implements System.Testability.ITestHandler. OnBeforeTestCaseRun resets per-case accuracy/turns/token accounting and opens the run-procedure output scope; OnAfterTestCaseRun records the case outcome. Consuming eval codeunits opt in via TestHandlers = "AIT Test Handler". - enumextension 149035 registers it under the platform TestHandler enum. - "AIT DD Current Case" now buffers the per-case identity/outcome (in memory, so it survives the function isolation rollback) and flushes one AIT Log Entry from the out-of-isolation OnAfterTestMethodRun seam (decision C8-A). Note: that seam is raised by the Test Runner - Mgt runner (AL Test Tool / Test Suite Mgt); the pure platform test-execution path (al CLI) does not raise it, so log persistence there is a tracked follow-up. - "AIT Test Suite Mgt.".AddDataDrivenLogEntry writes a per-case log entry with a session-scoped Run ID when there is no AIT suite; "AIT Test Run Iteration". IsRunningUnderAITSuite gates the handler off under an AIT suite to avoid double logging. - Register the new objects in the toolkit object permission set and document the TestHandlers step in the migration guide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AddDataDrivenLogEntry now calls AgentTestContextImpl.LogAgentTasks after inserting the AIT Log Entry, matching the classic AddLogEntry path so agent-eval task correlation is preserved for language-first evals running on the platform runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tion) Now that the platform runs ITestHandler hooks outside the per-function isolation scope, the handler can write the AIT Log Entry directly in OnAfterTestCaseRun and it persists past the case rollback. Remove the buffer/flush workaround: the pending-case state and the OnAfterTestMethodRun flush subscriber in AIT DD Current Case, and AIT DD Current Case is reduced to the dataset-lineage bridge + RunId + case-start baseline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
Could not find a linked ADO work item. Please link one by using the pattern 'AB#' followed by the relevant work item number. You may use the 'Fixes' keyword to automatically resolve the work item when the pull request is merged. E.g. 'Fixes AB#1234' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds language-first (
[TestDataSource]) data-driven support to the AI Test Toolkit so AIT evals can run on the platform test runner, coexisting with the classic app-based Eval Suite runner. Companion to platform (BC-Platform) and compiler (BC-DeveloperExperience) PRs.Changes (all under
src/Tools/AI Test Toolkit)src/DataDriven/):AIT Test Case Contextinterface (extendsITestContext, exposes the classicAIT Test Contextsurface);AIT DD Test Context(implements it);AIT Test Data Source(implements ITestDataSource, resolves the dataset from the AIT run context or the attribute literal, one context perTest Inputrow);AIT Test Handler(implements ITestHandler) + enum registration;AIT DD Current Case(dataset-lineage bridge + session run id + per-case baseline).AIT Log Entryper case directly inOnAfterTestCaseRun(relies on the platform change that runs handlers outside function isolation).AddDataDrivenLogEntry+IsRunningUnderAITSuitegate so the classic suite path and the platform-runner path never double-log.CodeUnit Metadata."Has Test Data Source") and added once (no per-row expansion);Language Firstflag on the eval line as an explicit override.Migration surface for consumers
A typical eval migrates with three declarations and no new objects:
[TestDataSource(Codeunit::"AIT Test Data Source", '<dataset>')], acontext: interface "AIT Test Case Context"parameter, andTestHandlers = "AIT Test Handler".Validation
Validated via
al runtestsagainst a local BC 29.0 server (provider + context fan-out, handler logging). Depends on the platform/compiler PRs to land for full behavior; theContext.Skip()credit-limit re-hook and streaming Harms provider are follow-ups once those ship.Draft.