Repository navigation
fix(opencode): redact short credential values from scan output - #745
Conversation
Signed-off-by: yashrajbasav <yashrajbasav@nvidia.com>
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Hi @yashrajp22, thank you for tightening the OpenCode tool's credential redaction and testing it on every success and failure path through truncation!
Value and readiness: The narrow goal is met: a configured 1-3 character value is now replaced on every model-visible path, and redaction still runs before truncation. Removing the length floor without bounding how short values are matched causes two regressions, and neither is tested:
- A one-character placeholder now defeats the existing
sk-key patterns, so most of a real key leaks. - A one-character placeholder also corrupts the JSON report the agent reads.
This is not ready yet.
Material findings
-
[Blocker]
.opencode/tools/skillspector_scan_lib.ts:47(with:58-61): Short configured values now split key-shaped tokens before thesk-ant-,sk-andNAME=valuepatterns get to them, because the configured-value pass runs first.- Example:
OPENAI_API_KEY=x(a common placeholder for local OpenAI-compatible servers), and a scanned skill whose flagged line containsapi_key="sk-proj-Q7wErTyxUiOp9AsDfGhJkL". Findingcode_snippettext reaches stdout verbatim, because the CLI'sredact_textonly scrubs URL credentials. - On
mainthe whole key becomes[REDACTED]. With this PR thexpass yieldssk-proj-Q7wErTy[REDACTED]UiOp….\bsk-…\bthen matches only up to the bracket, so the model sees[REDACTED][REDACTED]UiOp9AsDfGhJkL. - Any given letter or digit appears in about half of random 48-character keys and in most longer ones, so this is not a corner case.
Expected: apply the prefix and
NAME=valuepatterns before the configured-value matcher, or in the same pass. Add a regression with a short configured value plus an unrelatedsk-…/sk-ant-…key in the output. Reordering also closes the same, rarer split for configured values of 4+ characters. - Example:
-
[Blocker]
.opencode/tools/skillspector_scan_lib.ts:47: Literal substring replacement of 1-2 character values rewrites ordinary report text.- With
OPENAI_API_KEY=1,"risk_score": 12becomes"risk_score": [REDACTED]2and"start_line": 1becomes"start_line": [REDACTED]. The default--format jsonoutput is no longer valid JSON. - With
=e,truebecomestru[REDACTED]and"severity"becomes"s[REDACTED]v[REDACTED]rity". - The agent gets an unparseable or unreadable verdict, with nothing telling it why.
- The new tests pad with
o/eand check"before 7 after", so they never show what happens to ordinary text.
Expected: either keep a minimum length for global substring replacement, or redact values shorter than 4 characters only as standalone tokens (not adjacent to
[A-Za-z0-9_-]). Add a test that a representative JSON report stays parseable and readable when a one-character placeholder such asxoreis configured. - With
PIC tradeoffs: No provider in CREDENTIAL_ENV_NAMES issues keys shorter than 4 characters. A 1-3 character value is therefore almost certainly a placeholder, and if it were a real secret it would be trivially guessable. The PIC should decide whether such values are redacted at all (current main skips them) or only as whole tokens. Either way, the two blockers above must hold.
Verification and gaps:
- Output paths: I traced every model-visible path. These are success stdout and stderr (
formatSuccess), partial streams and themessagefallback (formatExecError), and the cancel, timeout and usage branches. All go throughredact()beforetruncate(), so the PR's coverage claim holds. - Not covered (pre-existing, all lengths): the
--outputreport file is written by the CLI and is not redacted. The literal matcher also misses encoded forms of a value: JSON-escaped, percent-encoded, base64, or line-wrapped. - Both regressions above were reproduced with a standalone Python model of the same pass order, using my own code, not the PR's. The regex semantics match JS for these inputs.
- Node tests were not executed, per policy. CI is
action_requiredwith no runs. - The four later commits are bot merges of
main. The PR's own diff is identical to the original commit, and the merged changes (#727, #731) touch unrelated files.git merge-treeagainst currentmainis clean. - #747 edits the same file in different hunks and merges cleanly with this PR, so either can land first.
Decision: Changes Requested (reviewed head 95c11bb3a3b5845fe77494a8b354c33a1bdf38d8)
Signed-off-by: yashrajbasav <yashrajbasav@nvidia.com>
…ries Signed-off-by: yashrajbasav <yashrajbasav@nvidia.com>
Signed-off-by: yashrajbasav <yashrajbasav@nvidia.com>
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Hi @yashrajp22, thank you for fixing both problems at the root and backing the fix with thorough overlap and JSON tests!
Value and readiness: Both blockers from the last review are fixed. Each has a regression test that fails on the previous head.
- Key splitting: configured values and key-shaped matches are now both found on the original text and merged into one redaction, so a short placeholder can no longer split an
sk-key. - Short values: values shorter than 4 characters match only as standalone tokens.
- JSON reports: when stdout is a JSON document, only string tokens are rewritten, so scores, line numbers and booleans stay intact.
The rework also closes three leaks that exist on main today (see Verification). Ready for final maintainer/PIC review. The finding and the PIC tradeoff below are optional.
Previous findings (review at 95c11bb. Since then: author commits 4ff755c5, 9604577a and 263b76e7, plus bot merges of main.)
- [Blocker] Short-value pass leaks
sk-key tails: Resolved.redact()now collects two span streams over the original text: configured values (.opencode/tools/skillspector_scan_lib.ts:53-63) and one combined key/assignment pattern (:64-68). It then merges overlapping or adjacent spans into a single[REDACTED](:86-111).- A value shorter than 4 characters must not touch
[A-Za-z0-9_-](:57).sk-keys use the same character set, so a short value can never match inside a key. - The last review's input was
OPENAI_API_KEY=xwithapi_key="sk-proj-Q7wErTyxUiOp9AsDfGhJkL"in a JSONcode_snippet. It now becomesapi_key="[REDACTED]". - The test at
tests/opencode/skillspector_scan_lib.test.ts:100-107pins this withxandQ7wE. Both cases fail at95c11bb, andQ7wEalso fails onmain.
- [Blocker] A one-character placeholder corrupts the JSON report: Resolved.
- If stdout parses as a JSON object or array, only string tokens are decoded, scrubbed and re-encoded (
:113-118). Everything else, including indentation, is left as is. - With
=1,=eor=x, anindent=2report like the CLI's stays valid JSON.risk_score: 12,start_line: 1andtrueare unchanged, and theseveritykey keeps its spelling. - The test at
:125-140pins this with1,eandx. At95c11bb, the1andecases produce invalid JSON.
- If stdout parses as a JSON object or array, only string tokens are decoded, scrubbed and re-encoded (
Material findings
- [Non-blocking]
.opencode/tools/skillspector_scan_lib.ts:64-79: key-shaped matches are now found on the original text. As a result, a configured value glued directly to a following key no longer creates the word boundary thatmainrelied on.- Example: with
NVIDIA_INFERENCE_KEY=nvapi-ABCD1234efgh, the textnvapi-ABCD1234efghsk-abcdef123456becomes[REDACTED]sk-abcdef123456. Likewise,nvapi-ABCD1234efghOPENAI_API_KEY=second-secretkeepssecond-secretvisible. - On
main: both are fully redacted, because the inserted[REDACTED]marker gives\bsk-and\b[A-Z]a boundary to match against. - Why optional: this needs a configured value of 4 or more characters that ends in a letter, digit or
_and is followed by the next secret with no separator. The configured value itself is always hidden, and I found no realistic SkillSpector output with this layout. - Suggested fix: also run
keyPatternover a copy of the text in which configured-value spans are replaced by a non-word character of the same length, and merge those spans as well. That keeps the code linear. A test such asredact("nvapi-ABCD1234efghsk-abcdef123456", { NVIDIA_INFERENCE_KEY: "nvapi-ABCD1234efgh" }) === "[REDACTED]"would pin it.
- Example: with
PIC tradeoffs: Standalone-token matching still rewrites standalone numbers in non-JSON formats when the placeholder is a digit.
- Example: with
OPENAI_API_KEY=1, the Markdown row| Score | 1/100 |becomes| Score | [REDACTED]/100 |, andSKILL.md:1becomesSKILL.md:[REDACTED]. - Not affected: JSON and SARIF output (the tool defaults to JSON), and letter placeholders such as
x. - Option: this is the "whole tokens" option from the last review. If the PIC would rather skip values shorter than 4 characters entirely, as
maindoes, the overlap and JSON fixes still stand on their own.
Verification and gaps:
-
Model: I wrote my own Python model of the new
redact(), not the PR's code. Pythonrein ASCII mode matches JS for these patterns on ASCII input. The model reproduces every expectation in the new tests. -
Exact coverage: I compared the merged output against a brute-force oracle on 100,000 random inputs, including self-overlapping values such as
abab. The oracle marks every occurrence of each configured value using its full surrounding context, plus the regex's own key matches. All 100,000 outputs matched. -
Comparison with
main: I tracked every original character throughmain's sequential passes. I used 200,000 random inputs, plus 200,000 inputs built around realistic configured values. Apart from finding 1, the PR never shows a character thatmainhides, except in two narrow cases that expose no key material:- a trailing
-after ansk-...token that contains an innersk-ant-(\bstops before trailing dashes, whilemain's first pass took them); - configured values that themselves contain
API_KEY,TOKEN,=or:.
Inserting a separator after each configured value removes every finding-1 difference.
- a trailing
-
Leaks on
mainthat this closes:- a configured value of 4 or more characters inside an unrelated key. For example,
Q7wEleavessk-proj-[REDACTED]rTyxUiOp9AsDfGhJkLonmain. - overlapping configured values. For example,
abcdpluscdefleaves[REDACTED]efonmain. - JSON output where an assignment or
sk-key starts a snippet line after an escaped\n. Onmain,\bfails after thenof\n, so"import os\nAWS_SECRET_ACCESS_KEY=..."and"...\nsk-abcdef..."pass through unredacted. The PR scrubs the decoded string and redacts both.
- a configured value of 4 or more characters inside an unrelated key. For example,
-
JSON validity: 30,000 random reports (compact and
indent=2) with configured values of 1-9 characters all stay parseable, with numbers, floats, booleans andnullunchanged. For--format jsonwithout--output, the CLI prints only the report to stdout (cli.py:371). Progress output and transitive-scan warnings go to stderr. The JSON branch therefore applies in the tool's default flow. -
Display boundary, quotes and URLs:
- I placed
sk-keys, configured values and short values across the 12,000-character display limit at a range of offsets. No partial prefix stayed visible, because redaction still runs before truncation on every path. - Short values are redacted inside quotes, brackets and URLs (
https://x:x@host/x?k=x) and after:or=. Insidex86,xyz,x-yandx_ythey are left alone.
- I placed
-
Linear time: my model's run time doubles when the input doubles (ratios 1.96-2.07 between 200,000 and 400,000 characters). The inputs were:
- long word runs;
sk-followed by a run of dashes;NAME=followed by spaces;- repeated
API_KEnear-misses; - the value
aaaaover a run ofa, which hits the lookahead at every position; - a repeated standalone
x; - JSON with one very large string, and JSON with many small strings.
-
Merged
main:- The #747 timeout code is intact (
:9,:228,:578,:613,:622). - The
geminiprovider added in #496 uses Vertex AI credentials (GOOGLE_CLOUD_PROJECT), not an API-key variable, soCREDENTIAL_ENV_NAMESneeds no new entry. git merge-treeagainstmainis clean, and the patch-id is identical at263b76e7and at the head. No other open PR touches these files.
- The #747 timeout code is intact (
-
CI: on
263b76e7, "OpenCode TypeScript Tests", lint and DCO passed, and test-unit was still running. The head81ab513is a bot merge with CIaction_required. -
Not run: contributor tests, per policy.
-
Review thread: you replied to my inline thread and resolved it. Both parts of it are fixed.
Decision: Approved (reviewed head 81ab5137cad55510619dce4cb8e5ed0b4247426c)
Signed-off-by: yashrajbasav <yashrajbasav@nvidia.com>
Signed-off-by: yashrajbasav <yashrajbasav@nvidia.com>
Signed-off-by: yashrajbasav <yashrajbasav@nvidia.com>
|
Thanks for spotting this! Credentials directly after a configured value are now caught on the original text, so overlapping values cannot split another key. I also made assignment detection scan each name once, which keeps long uppercase output from slowing it down. All 56 Node tests pass, including the adjacent-key, JSON, overlap, and runtime cases. |
Configured credentials shorter than four characters were omitted from OpenCode output redaction. Redact those values as whole tokens so common placeholders do not corrupt words, risk scores or flags. JSON output is scrubbed inside strings while preserving numeric and boolean metadata.
Merge overlapping configured-value and key-shaped matches on the original text so partial or adjacent matches cannot expose another credential. Assignment detection consumes each name once to avoid repeatedly scanning long uppercase output. Redaction runs before truncation on model-visible success/error paths. The separate report file written by --output is outside this redactor.
Validation: all 56 OpenCode helper tests pass, including short placeholders, adjacent full keys and assignments, JSON metadata, punctuation-bearing and overlapping credentials, truncation and error paths. The 128 KiB repeated-uppercase regression completed in about 10 ms in the isolated test container.