Add PR-triggered documentation drift checks#90
Merged
Conversation
- Documents automated PR-triggered doc drift detection system - GitHub Actions workflow with OWNER-only comment trigger (/check-docs) - Pi agent skill with 5-phase decision tree (intent extraction → doc search → cross-reference → coverage assessment → report) - Intent-based analysis (not per-file) with multi-term rg search strategy - False negative guard: multiple search term sets before concluding absence - Structured PR comment output with copy-pasteable agent fix prompt - Two reference files: user-facing-criteria.md and comment.md - Environment-aware: GitHub context posts comment, local context prompts user - Covers both drift detected and undocumented user-facing change paths
- Remove 2-4 intent cap: agent extracts all behavioral changes, no ceiling - Remove rg tool prescription: agent uses whatever search tools are available - Replace numeric false negative guard with coverage-based stop signal - Frame user-facing criteria list as illustrative, not exhaustive - Rewrite fix prompts to be direct and concise (no redundant review step) - Switch output model to one file per behavioral change, posted as separate PR comments; each finding gets its own fix prompt - Replace 'intent' with 'behavioral change' in all user-facing output - Add language note to comment.md: never use the word 'intent' in output - Update edge cases table to match new per-change comment model
6 tasks: Pi skill, two reference files, GitHub Actions workflow, setup notes, and end-to-end smoke test. Flags two TODOs that need Pi CLI syntax confirmation before first CI run.
- Gate on OWNER author_association, react with 👀 on the triggering comment, and post a failure comment if the check errors - Guide a Pi agent through 5 phases: extract behavioral changes from the diff, search docs/ with a multi-term strategy, cross-reference claims, assess user-facing coverage, write per-change report files - Add VectorLint-specific user-facing criteria and output format references covering drift, undocumented changes, and clean pass in both GitHub Actions and local contexts
- Expand documentation search targets from docs/*.mdx only to also cover README.md, CLAUDE.md, and AGENTS.md
- split authorization into its own job so the doc drift pipeline only runs once the owner check passes - replace inline workflow Python with a checked-in TypeScript helper for building the Pi prompt - confirm the Pi CLI invocation and move the doc-drift skill under .agents/skills
- add AGENTS.md so harnesses that prefer AGENTS instructions read the same repository guidance - replace CLAUDE.md with a symlink to AGENTS.md to keep Claude Code compatibility without duplicating content
- switch the production doc drift workflow from Anthropic envs to Amazon Bedrock provider wiring and explicit model selection - add an act-only workflow harness under tests/workflows so branch diffs can be exercised locally without enabling another GitHub Actions workflow - add a .secrets example and ignore the real .secrets file to make local workflow testing easy without risking committed credentials
- rename the model secret to PI_MODEL in the production and local doc drift workflows so it matches the existing Pi env naming - remove the unused GitHub CLI installation from the act-only test workflow because the local harness builds diffs with git and uploads artifacts instead of calling GitHub APIs
- run the doc-drift workflow from separate main and PR checkouts, and pass the diff path into the agent instead of embedding the full diff in the prompt - split doc-drift output templates into GitHub and non-GitHub structures, including the shorter no-drift comment and environment-specific detail formatting - persist local workflow test artifacts under artifacts/doc-drift and document the act-based test flow for iterating on the workflow locally
|
Warning Review limit reached
More reviews will be available in 10 minutes and 30 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (7)
📝 WalkthroughWalkthroughIntroduces a complete "doc drift" checking system for VectorLint. A ChangesDoc Drift Checker
Sequence Diagram(s)sequenceDiagram
participant Owner as PR Owner
participant GHA as GitHub Actions
participant Pi as Pi Agent (Bedrock)
participant Docs as docs/*.mdx
participant PR as Pull Request
Owner->>GHA: Comment "/check-docs"
GHA->>GHA: Validate OWNER association
GHA->>GHA: gh pr diff → diff.patch
GHA->>GHA: build-doc-drift-message.ts → message.txt
GHA->>Pi: pi run --skill doc-drift --message message.txt
Pi->>Pi: Extract behavioral changes from src/ diff
Pi->>Docs: Search for each behavioral change
Docs-->>Pi: Matched doc sections (or no match)
Pi->>Pi: Cross-reference: are claims invalidated?
Pi->>Pi: Check user-facing-criteria.md for unmatched changes
Pi-->>GHA: Write .doc-drift-N.md per finding
GHA->>PR: Post one comment per report file
GHA-->>PR: Post failure comment on error
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
🧹 Nitpick comments (3)
docs/plans/2026-06-17-doc-drift-checker.md (1)
257-257: 💤 Low valueAdd language specifiers to code blocks.
Code blocks on lines 257, 495, 509, and 539 lack language specifiers. Use
bash,markdown, or appropriate language tags.Also applies to: 495-495, 509-509, 539-539
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/plans/2026-06-17-doc-drift-checker.md` at line 257, The code blocks on lines 257, 495, 509, and 539 are missing language specifiers in their markdown syntax. For each code block that currently uses triple backticks without a language identifier (just ```), add the appropriate language specifier immediately after the opening backticks (such as bash for shell commands, markdown for markdown content, or other relevant language tags). This improves syntax highlighting and readability in the documentation.Source: Linters/SAST tools
docs/superpowers/specs/2026-06-17-doc-drift-design.md (1)
24-24: 💤 Low valueAdd language specifiers to code blocks.
Five code blocks (lines 24, 77, 112, 297, 325) lack language specifiers. Use
yaml,text, ormarkdownas appropriate to aid syntax highlighting.Also applies to: 77-77, 112-112, 297-297, 325-325
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/superpowers/specs/2026-06-17-doc-drift-design.md` at line 24, Add language specifiers to the five code blocks in the document that currently lack them. For each code block starting at the locations indicated (lines 24, 77, 112, 297, 325), modify the opening fence from triple backticks with no language identifier to include the appropriate language specifier such as yaml, text, or markdown (e.g., change ``` to ```yaml or ```text or ```markdown as appropriate for the content). This will enable proper syntax highlighting for each code block.Source: Linters/SAST tools
tests/workflows/docs-drift-test.yml (1)
35-56: Pin workflow CLI tool versions for reproducible local test runs.Lines 36 and 55 currently pull latest tool versions at runtime, which can make manual workflow tests drift unexpectedly across runs. Pin
@earendil-works/pi-coding-agentto a specific version (current:0.79.6) and addtsxversion pinning withnpx --yes tsx@<PINNED_VERSION>(current:4.22.4).Suggested change
- - name: Install Pi - run: npm install -g `@earendil-works/pi-coding-agent` + - name: Install Pi + run: npm install -g `@earendil-works/pi-coding-agent`@0.79.6 @@ - npx tsx .github/scripts/build-doc-drift-message.ts "$GITHUB_WORKSPACE/.doc-drift-input.diff" /tmp/pi-message.txt + npx --yes tsx@4.22.4 .github/scripts/build-doc-drift-message.ts "$GITHUB_WORKSPACE/.doc-drift-input.diff" /tmp/pi-message.txt🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/workflows/docs-drift-test.yml` around lines 35 - 56, Pin the npm package versions in the workflow to ensure reproducible test runs. In the Install Pi step, add the version specifier to the npm install command for `@earendil-works/pi-coding-agent` by appending `@0.79.6` to the package name. In the Build agent message step, replace the npx tsx command with npx --yes tsx@4.22.4 to pin the tsx tool to version 4.22.4 instead of using the latest available version at runtime.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/scripts/build-doc-drift-message.ts:
- Line 3: The const variable declarations on lines 3, 9, and 15 use camelCase
naming (diffPath, workspace, message) which violates the ESLint naming
convention rule requiring UPPER_CASE for const variables. Rename diffPath to
DIFF_PATH, workspace to WORKSPACE, and message to MESSAGE in their respective
const declarations, then update all references to these variables throughout the
file to use the new UPPER_CASE names to comply with the project's linting rules.
In @.github/workflows/doc-drift.yml:
- Line 27: Pin all GitHub Actions references in the workflow file to specific
commit hashes instead of version tags for security compliance. Replace the six
unpinned action references (actions/github-script@v7 on lines 27, 57, and 69;
actions/checkout@v4 on lines 80 and 87; and actions/setup-node@v4 on line 94)
with their full commit SHA hashes by visiting each action's releases page on
GitHub and copying the complete commit hash for the desired version tag (e.g.,
actions/checkout@a5ac7e51b41094c5145c72f7b3c3d9be72cb94a8).
- Around line 79-84: Add the `persist-credentials: false` parameter to both
checkout steps in the workflow. For the first checkout step named "Checkout main
branch" (which uses actions/checkout@v4 with ref pointing to the default
branch), add `persist-credentials: false` as a new line in the with section.
Similarly, add the same parameter to the second checkout step that is also
mentioned in the comment (lines 86-91). This ensures the GitHub token is not
persisted as a git credential after checkout.
- Around line 16-20: The check-docs job declares pull-requests permission as
read, but the job uses gh pr comment to post comments to pull requests, which
requires write permission. Update the pull-requests permission in the
permissions block of the check-docs job from read to write. This ensures the
workflow has the necessary permissions to execute the gh pr comment command
successfully.
In `@AGENTS.md`:
- Line 154: The line describing Anthropic models incorrectly includes "Codex
models" which are actually OpenAI's model family. Remove the "Codex models"
reference from the Anthropic line and ensure it correctly attributes only Claude
(Opus, Sonnet, Haiku) to Anthropic. Update the Anthropic entry to accurately
reflect that Anthropic produces the Claude model family, not Codex.
In `@docs/superpowers/specs/2026-06-17-doc-drift-design.md`:
- Around line 347-366: The documentation contains a duplicated "Local context
(no GITHUB_ACTIONS)" section that appears twice consecutively with identical
content, creating confusion about whether there are differences in behavior
between them. Remove the second occurrence of the entire "Local context (no
GITHUB_ACTIONS)" block that repeats the instructions about printing behavioral
changes, prompting the user for documentation updates, and handling responses -
retain only the first instance of this section to eliminate the duplication.
In `@tests/workflows/README.md`:
- Around line 6-7: The sentence in the workflow test rationale contains an
awkward and grammatically incorrect phrase "create dirty commit on iteration".
Reword this phrase to be grammatically correct and more clearly convey that
running tests locally avoids cluttering the repository history with unnecessary
commits during development iteration. The corrected sentence should flow
naturally and be easily understood by contributors reading the README.
---
Nitpick comments:
In `@docs/plans/2026-06-17-doc-drift-checker.md`:
- Line 257: The code blocks on lines 257, 495, 509, and 539 are missing language
specifiers in their markdown syntax. For each code block that currently uses
triple backticks without a language identifier (just ```), add the appropriate
language specifier immediately after the opening backticks (such as bash for
shell commands, markdown for markdown content, or other relevant language tags).
This improves syntax highlighting and readability in the documentation.
In `@docs/superpowers/specs/2026-06-17-doc-drift-design.md`:
- Line 24: Add language specifiers to the five code blocks in the document that
currently lack them. For each code block starting at the locations indicated
(lines 24, 77, 112, 297, 325), modify the opening fence from triple backticks
with no language identifier to include the appropriate language specifier such
as yaml, text, or markdown (e.g., change ``` to ```yaml or ```text or
```markdown as appropriate for the content). This will enable proper syntax
highlighting for each code block.
In `@tests/workflows/docs-drift-test.yml`:
- Around line 35-56: Pin the npm package versions in the workflow to ensure
reproducible test runs. In the Install Pi step, add the version specifier to the
npm install command for `@earendil-works/pi-coding-agent` by appending `@0.79.6` to
the package name. In the Build agent message step, replace the npx tsx command
with npx --yes tsx@4.22.4 to pin the tsx tool to version 4.22.4 instead of using
the latest available version at runtime.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 08f522aa-9461-4211-ac57-5f15ee15375c
📒 Files selected for processing (15)
.agents/skills/doc-drift/SKILL.md.agents/skills/doc-drift/references/comment.md.agents/skills/doc-drift/references/user-facing-criteria.md.github/scripts/build-doc-drift-message.ts.github/workflows/doc-drift.yml.gitignore.secrets.exampleAGENTS.mdCLAUDE.mdCLAUDE.mddocs/plans/2026-06-17-doc-drift-checker.mddocs/superpowers/specs/2026-06-17-doc-drift-design.mdtests/workflows/README.mdtests/workflows/docs-drift-test.ymltsconfig.json
- rename local bindings in the doc-drift message builder to match the repo naming convention - keep the workflow prompt builder behavior the same while allowing eslint --fix and the pre-commit hook to pass cleanly
- pin GitHub Action SHAs and workflow tool versions so doc-drift runs stay reproducible and less exposed to upstream drift - correct the Anthropic provider description and improve the local workflow test wording for contributors - remove tracked planning artifacts from docs/plans and docs/superpowers, then ignore those directories so local planning files stay out of version control
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Our documentation drifts from product behavior as we build, creating a poor experience for people evaluating VectorLint and for existing users. As a small team, we need a way to catch that drift during PR review and either address it immediately or follow up on it intentionally.
What this PR covers
Behavior impact
/check-docs.Future improvements
/check-docs.How to test / verify
Manual verification
Run:
Confirm
artifacts/doc-drift/contains:pr.diffpi-message.txt.doc-drift-*.mdfiles when findings are producedOn a GitHub PR, comment
/check-docsas the repo owner and verify:eyesreactionSummary by CodeRabbit
New Features
/check-docsPR comments, identifying when code changes invalidate or lack corresponding documentation updates.Documentation
Tests
Chores