Repository navigation
Clarify the references prompt and report the model's LSP calls - #35
Merged
Merged
Conversation
- The rb6 prompt asks for the callers a return-type change would affect and says not to change code. The old wording read as a request to make the change, and the model stopped to ask what the new type was. - A missing-lookup failure now lists the LSP calls the model actually made, and when there were none, suggests checking the operation enum. A test covers the no-call case. - The recorded fixture carries the new prompt as the tester logs it, shell-quoted.
rb6's aliased-caller-and-declaration checked a subset of affected-callers-across-files, and bn2's definition-follows-edited-import was the first half of successive-document-versions. Neither caught a mistake the remaining case misses. Removing them saves two live-model runs per full test. The fixture drops both blocks, and bn2's timeout matches rb6's two-case budget.
- ql9 runs 3 cases instead of 5. Stale and unversioned reports share one server config, and the no-report case also shifts the definition below a comment. Each pair checks independent behaviour, so one run covers both, and 3 cases fit the 190s timeout. - bn2's write-before-open prompt said "look up", and with findReferences available the model chose it. The prompt now names goToDefinition, and the restore step says "definition lookup". - The fixture follows the new case names and prompts, and the shifted definition recording test uses the merged case.
LSP results must name each location's relative file and 1-based position, in any wording. Diagnostics are no longer read from Write's result: every current diagnostic the server published, for any open file, must reach the model, and stale reports must not. The fixture server re-checks every open document after a change, so an unwritten caller's error is reported. The reference agent, both smoke tests and the fixture follow the new formats, and the Claude Code smoke test pins them on 2.1.318.
A notify call returns before the fixture server reads the message, and the server checks didOpen against the file on disk. The tests edited the file right after didOpen, so the server sometimes saw the edited text and exited. They now make one request after didOpen, which the server answers only after handling it.
The post-edit goToDefinition made the model count columns in code it had just written, and a correct agent failed whenever it miscounted. ql9 now requires only the post-edit findReferences, whose result must match the edited files. Synchronization is still checked through the server's text and the follow-up references. The migration also no longer requires a Read before each Write, since the prompt never asks for one; the final files and tests grade the outcome.
|
Triggered a Github Actions job to update fixtures. |
Protocol errors such as a duplicate didOpen stopped the server without a log entry, and the stage then reported only that the program exited. Serve now logs every error it returns as a failure event, and the stage reports that reason first.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Medium Risk
Changes LSP grading rules and case prompts, which can fail previously passing learner submissions even though behavior is confined to the tester and reference scenarios.
Overview
Pins Claude Code to 2.1.318 in CI, the Makefile, README, and smoke scripts so compatibility checks match the product version under test.
Reworks LSP stage grading to match how Claude Code 2.1.318 talks to the model: LSP tool results must name relative paths and 1-based line:character text (raw server JSON is rejected via new
checkFormattedResult). Diagnostics are no longer required inside each Write’s structured JSON; instead the checker scans the full conversation for every currentpublishDiagnosticsissue (file, 1-based position, message), rejects stale versioned reports, and expects delivery like<new-diagnostics>appended to a later tool result. The fixture server and reference agent mirror that (cross-file republication after edits, formatted LSP strings,new_diagnostics()on the agent loop).Trims and refocuses cases: rb6 drops the minimal-project case and uses a prompt that asks for affected callers without changing code; ql9 drops the post-edit goToDefinition requirement and merges several diagnostic variants into one stale + unversioned + shifted definition case; bn2 loses one case; migration no longer grades pre-edit Reads or invoice_total call position on follow-up definition. Missing-lookup failures now list the model’s actual LSP calls and hint at the operation enum when none were made.
Docs, recorded fixtures, Pyright/Claude smoke assertions, and transport
opened()synchronization are updated accordingly; bn2 stage timeout is lowered to 145s.Reviewed by Cursor Bugbot for commit f06e985. Bugbot is set up for automated code reviews on this repo. Configure here.