Skip to content

Clarify the references prompt and report the model's LSP calls - #35

Merged
yeswanth merged 7 commits into
mainfrom
lsp-stage-description-review
Nov 5, 2026
Merged

yeswanth merged 7 commits into
mainfrom
lsp-stage-description-review

Conversation

@yeswanth

@yeswanth yeswanth commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor
  • The rb6 prompt asks for the callers a return-type change would affect and says not to change code. The old wording read as a request to make the change, and the model stopped to ask what the new type was.
  • A missing-lookup failure now lists the LSP calls the model actually made, and when there were none, suggests checking the operation enum. A test covers the no-call case.
  • The recorded fixture carries the new prompt as the tester logs it, shell-quoted.

Note

Medium Risk
Changes LSP grading rules and case prompts, which can fail previously passing learner submissions even though behavior is confined to the tester and reference scenarios.

Overview
Pins Claude Code to 2.1.318 in CI, the Makefile, README, and smoke scripts so compatibility checks match the product version under test.

Reworks LSP stage grading to match how Claude Code 2.1.318 talks to the model: LSP tool results must name relative paths and 1-based line:character text (raw server JSON is rejected via new checkFormattedResult). Diagnostics are no longer required inside each Write’s structured JSON; instead the checker scans the full conversation for every current publishDiagnostics issue (file, 1-based position, message), rejects stale versioned reports, and expects delivery like <new-diagnostics> appended to a later tool result. The fixture server and reference agent mirror that (cross-file republication after edits, formatted LSP strings, new_diagnostics() on the agent loop).

Trims and refocuses cases: rb6 drops the minimal-project case and uses a prompt that asks for affected callers without changing code; ql9 drops the post-edit goToDefinition requirement and merges several diagnostic variants into one stale + unversioned + shifted definition case; bn2 loses one case; migration no longer grades pre-edit Reads or invoice_total call position on follow-up definition. Missing-lookup failures now list the model’s actual LSP calls and hint at the operation enum when none were made.

Docs, recorded fixtures, Pyright/Claude smoke assertions, and transport opened() synchronization are updated accordingly; bn2 stage timeout is lowered to 145s.

Reviewed by Cursor Bugbot for commit f06e985. Bugbot is set up for automated code reviews on this repo. Configure here.

- The rb6 prompt asks for the callers a return-type change would affect
  and says not to change code. The old wording read as a request to make
  the change, and the model stopped to ask what the new type was.
- A missing-lookup failure now lists the LSP calls the model actually
  made, and when there were none, suggests checking the operation enum.
  A test covers the no-call case.
- The recorded fixture carries the new prompt as the tester logs it,
  shell-quoted.
rb6's aliased-caller-and-declaration checked a subset of
affected-callers-across-files, and bn2's definition-follows-edited-import
was the first half of successive-document-versions. Neither caught a
mistake the remaining case misses. Removing them saves two live-model runs
per full test. The fixture drops both blocks, and bn2's timeout matches
rb6's two-case budget.
- ql9 runs 3 cases instead of 5. Stale and unversioned reports share one
  server config, and the no-report case also shifts the definition below
  a comment. Each pair checks independent behaviour, so one run covers
  both, and 3 cases fit the 190s timeout.
- bn2's write-before-open prompt said "look up", and with findReferences
  available the model chose it. The prompt now names goToDefinition, and
  the restore step says "definition lookup".
- The fixture follows the new case names and prompts, and the shifted
  definition recording test uses the merged case.
LSP results must name each location's relative file and 1-based position,
in any wording. Diagnostics are no longer read from Write's result: every
current diagnostic the server published, for any open file, must reach the
model, and stale reports must not. The fixture server re-checks every open
document after a change, so an unwritten caller's error is reported.

The reference agent, both smoke tests and the fixture follow the new
formats, and the Claude Code smoke test pins them on 2.1.318.
A notify call returns before the fixture server reads the message, and the
server checks didOpen against the file on disk. The tests edited the file
right after didOpen, so the server sometimes saw the edited text and exited.
They now make one request after didOpen, which the server answers only after
handling it.
The post-edit goToDefinition made the model count columns in code it had
just written, and a correct agent failed whenever it miscounted. ql9 now
requires only the post-edit findReferences, whose result must match the
edited files. Synchronization is still checked through the server's text
and the follow-up references.

The migration also no longer requires a Read before each Write, since the
prompt never asks for one; the final files and tests grade the outcome.
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

Triggered a Github Actions job to update fixtures.

Protocol errors such as a duplicate didOpen stopped the server without a
log entry, and the stage then reported only that the program exited. Serve
now logs every error it returns as a failure event, and the stage reports
that reason first.
@yeswanth
yeswanth merged commit 9fe518e into main Nov 5, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant