Skip to content

Run the xUnit v3 test executable directly instead of through dotnet test (~45% faster) - #330

Merged
ErikSchierboom merged 6 commits into
mainfrom
csharp-direct-xunit
Nov 18, 2026
Merged

ErikSchierboom merged 6 commits into
mainfrom
csharp-direct-xunit

Conversation

@iHiD

@iHiD iHiD commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Hello 🙂 I'm working with Claude to try and get our tooling faster.

This PR makes this project around 45% faster, and it's the first of three that together make it around 5x faster. I've run it in production to benchmark things in the real world against the most recent few hundred solutions, so I'm confident that it works robustly. However, I've not written the code. I have tried my hardest to get Claude to write sensible messages and comments that explains stuff, but I'd ask for your patience in the fact I'm sending AI code to you. Everything below here is from your friendly neighbourhood LLM...


This is the first of three PRs. Merge them in order: this one (#330), then #331, then #332.

This PR includes Erik's commit that removes xUnit v2, so every exercise uses xUnit v3.

What changed

  • The runner now builds with dotnet build and runs the test executable directly. Before, every run called dotnet test, which builds the project and then starts VSTest and a test host to run the tests. An xUnit v3 test project builds to an executable that can run its own tests, so the runner now skips VSTest and the test host.
  • The runner now reads the results from xUnit's XML report, in place of VSTest's TRX file. TestResultParser.cs reads each test element of the report. The rest of the code is unchanged.
  • xUnit's report escapes some characters in failure messages (for example \n for a newline). When the runner reads the report, it converts the escapes back into the original characters.
  • The runner always adds the attributes that capture console output. It used to add them only for xUnit v3 projects.

Failure messages

None of the messages in the golden tests change. A few rare messages will change, because VSTest used to rewrite them and the runner now shows what the code produced:

  • If a message has a control character, VSTest wrote it as the text \u0001. The message now has the character itself.
  • If a message has a lone carriage return, it now shows as a newline.
  • If a message has ANSI colour codes, VSTest removed them. The message now keeps them.

If the test executable crashes, or the student's code calls Environment.Exit, xUnit doesn't write its report, and the run will end with status error and no tests. dotnet test gave the same result for a stack overflow. For Environment.Exit, dotnet test reported a pass covering only the tests that had finished. A failure message containing ]]> (very rare) will also give error, because xUnit fails while it writes the report.

This has been tested in production, and has reduced the median run from 9.0s to 6.2s.

🤖 Generated with Claude Code

@ErikSchierboom

Copy link
Copy Markdown
Member

The changes are slightly odd , but I think what could help is if I first remove the xUnit v2 code (that has been around since February 2025), as that causes a lot of friction. You are also allowed to change the message field in the expected_results.json file

@ihid-assistant

Copy link
Copy Markdown

Thanks Erik, that makes sense. Removing the xUnit v2 code first would make this a lot simpler, and knowing the messages in expected_results.json can change helps too: the runner would no longer need to convert xUnit's escaping into VSTest's form, so most of XunitXmlParser.cs and the special cases around control characters can go.

Once your change is in, I'll rebase this stack onto it and simplify the three PRs.

@ErikSchierboom

Copy link
Copy Markdown
Member

Okay, just pushed the changes

ErikSchierboom and others added 2 commits November 18, 2026 18:18
The runner now reads the test results from xUnit's XML report, in place of VSTest's TRX file.

Co-authored-by: iHiD <286476+iHiD@users.noreply.github.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ErikSchierboom

Copy link
Copy Markdown
Member

You can redo the pr changes now

@ihid-assistant

Copy link
Copy Markdown

Thanks Erik! I'm reworking the stack on top of your changes now and will update all three PRs once it's checked.

@ihid-assistant
ihid-assistant Bot force-pushed the csharp-direct-xunit branch from 9c0086a to e725cbb Compare November 18, 2026 10:52
@ihid-assistant ihid-assistant Bot changed the title Run the xUnit v3 test executable directly instead of through dotnet test (~30% faster) Run the xUnit v3 test executable directly instead of through dotnet test (~45% faster) Nov 18, 2026
@ihid-assistant

Copy link
Copy Markdown

Thanks Erik, the stack is now reworked on top of your xUnit v2 removal, and I've updated all three descriptions.

Your commit comes first in #330, unchanged. With v2 gone and the messages free to change, #330 no longer needs the dotnet test path, the TRX reader or XunitXmlParser.cs, so it went from +221/−28 lines to +73/−85. #332 now has one test-run routine and one fallback. The golden tests pass, and the 50 recent solutions and 186 example solutions give the same results as main, messages included.

A few rare cases now give different results, and the #330 description lists them. One is Environment.Exit in a solution, which now gives error where dotnet test reported a false pass.

@ErikSchierboom
ErikSchierboom merged commit a92e483 into main Nov 18, 2026
1 check passed
@ErikSchierboom
ErikSchierboom deleted the csharp-direct-xunit branch November 18, 2026 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants