feat(python): classify executable source surfaces

[chrisknvidia]

Classify Python execution surfaces consistently from raw bytes across .py, .pyw, supported encodings and extensionless Python/uv launchers. Carry classification through nested and transitive analysis, inventory, inspection ledgers and report generation; ambiguous inputs retain incomplete coverage.

Review fixes keep uninvoked global/nonlocal stores, annotation-only declarations, unrelated attribute stores and same-slot assignments from suppressing lexical HIGH. Explicit called-slot replacement has a bounded lifetime: cached-module changes are tracked across ordinary imports, while unknown eager effects discard replacement certainty. Native callable captures, cross-API assignments and native values wrapped in RHS expressions cannot establish replacement proof. The legacy first-statement direct shadow control remains bounded to a single completed store with no native receiver/Popen reference and no later eager effect. Receiver binding, report ownership and cached slot state remain separate. Prose output tests retain the established occurrence-specific IDs and snippets while checking the shared match fingerprint.

This branch includes the still-open #577 and #578 changes. #576 is merged.

Validation of this revision: 826 affected analyzer/graph/input cases passed (one Linux-only O_PATH case skipped on macOS); 51 fresh installed-wheel CLI scans passed across positive/negative source cases, real reports and strict exits; four additional .pyw/env/uv/nested-archive source-surface scans passed. Ruff lint/format for src/tests and diff checks passed. Static verification uses real native analyzer/graph execution and an installed wheel with --no-llm; remote provider inference, production deployment and other operating systems were not exercised. Earlier broad-suite results were from older heads and do not establish a green full suite for this update. Hosted CI and human re-review remain required.

[rng1995]

Reviewed exact draft head 266d42e050493dad869fe50be420ace97914ea82. I found no additional blocker in the focused Python execution-surface range (d856d88..266d42e): 1,118 affected tests passed with 1 skip, 732 current-main merged-tree regressions passed, and lint/format/diff checks are clean.

I am requesting changes because the current combined tree still contains the confirmed #576/#577 blockers: a three-row ledger cap drops a second distinct fatal fact, and a definitely true bound shell= value evades TM1 when a later argument is effectful. This PR also inherits draft #578 and explicitly requires the dependency stack to land, rebase onto current main, fresh exact-head CI, and current-head review.

Please propagate the dependency fixes and rebase. If the focused range remains semantically unchanged and CI stays green, I found it otherwise suitable for approval.

[chrisknvidia]

@rng1995 The final #578/#577 stack is included, conflicts are resolved, and the bare-uv/parser-limit fixes remain intact. PTAL.

[rng1995]

The focused #579 bare-uv and execution-type parse-limit changes look correct, but this stacked head inherits the exact #577 receiver-invalidation blocker: an effectful argument after shell= may replace the trusted subprocess receiver, yet later proxy calls are still classified as subprocess and receive false-positive TM1 findings. Please update the stack after #577 clears trusted_names on this effectful path and add the regression. The current head is otherwise conflict-free and its affected/integration suites are clean.

[rng1995]

[SkillSpector Review]

Changes requested on exact head 59a8d9fd60b150b27457254979c256bc80b82e37 for a confirmed remaining issue inherited from #577.

The earlier effectful-argument fix is present: direct subprocess calls now invalidate receiver trust after unsafe arguments, including expression, assignment, and annotated-assignment paths. However, an ordinary effectful function call can still replace the receiver without invalidating that trust; a later call through the replacement object is then reported as HIGH TM1. The inline comment provides the source-traced fixture and expected correction. Please propagate the #577 correction through this stack and cover this case.

Scope: I inspected the prior review history, current dependency fix, receiver-trust collector and both scan passes, its caller integration, tests for the direct-argument correction, and current checks. This is a focused blocking review, not approval or certification of the entire large combined stack. The remaining combined diff still needs complete current-head assessment after the dependency issue is addressed. Six hosted checks pass, but passing checks do not establish this untested receiver semantic.

No contributor-provided code or tests were executed locally.

[rng1995]

[SkillSpector Review]

Re-reviewed the previous receiver-invalidation finding on exact head 3f2f7f70eeaa1cafc70792ef29e00ebd15b8d7de, including its current shared implementation, forward/prepass effects, caller/ownership integration, all prior reviews/replies, and relevant regression tests.

The exact standalone-call examples and the earlier effectful-argument ordering fixes are present. However, the required receiver-safety correction is still incomplete: a generic effectful call nested in a tuple/list RHS leaves the receiver trusted, and a subsequent fresh true binding makes a proxy call a false-positive HIGH TM1. The companion implementation is identical to the current #578 version at this point. The inline continuation describes the source-traced case and required regression coverage.

All six hosted checks pass, but their generic-call tests cover only outer ast.Call values. Please fix this in the shared implementation and propagate the correction through the stack.

Review scope remains a focused blocking re-review of the dependency and its integration, not full clearance of the 31-file, 11,690-added-line combined execution-surface stack. Complete current-head assessment of decoding/classification, provider boundaries, nested/transitive analysis, reporting, and inherited window/identity changes remains required before approval after the blocker is fixed. No contributor code/tests were executed locally; no merge was performed.

[rng1995]

[SkillSpector Review]

Hi @chrisknvidia, thank you for the quick turnaround on the nested receiver-invalidation finding and for the much broader regression matrix!

Value and readiness: The nested tuple/list receiver finding is fixed. The tighter receiver-trust model that came with it lives in the shared #577/#578 module. It now silences TM1 shell=<name> findings that main reports today in ordinary script layouts, which reopens the issue #475 evasion class. This needs a fix in the shared implementation before the stack is ready. Also, the latest commit a4d8525 is not a pure main merge (details below), and #577 is still open with changes requested.

Previous findings:

• Nested tuple/list/annotated RHS receiver invalidation (last review, inline thread): Resolved. _expression_preserves_receiver_trust (static_python_shell_truthiness.py:780-835) now rejects any eager expression that is not passive or a passive direct subprocess call. It recurses into tuples, lists and comprehensions. The forward scan (_scan_assignment, AnnAssign, Expr, Assert) and the _advance_trusted_names prepass both use it. Covered by test_nested_generic_call_invalidates_receiver_trust[tuple-rhs|list-rhs|annotated-tuple-rhs], its called-function variant and test_eager_container_expression_invalidates_*. Real positives remain in test_direct_subprocess_call_remains_detected.
• Earlier findings (inherited #576/#577 ledger-cap and later-effectful-argument blockers, effectful arguments after shell=, standalone generic calls): Resolved in earlier rounds and still in place at this head.
• Rebase and fresh CI: Resolved for main (main merged at 8831219, 6/6 checks green). #577 remains open.

On your inline reply: I confirmed the fix and the new regressions at a4d8525. However, a4d8525 is titled "Merge main" but carries author changes beyond conflict resolution (git show --remerge-diff a4d8525): the truthiness module grows from 1,381 to 1,854 lines and is now byte-identical to #578 head 68e22d0, test_tool_misuse_python_ast.py grows from 842 to 1,484 lines, a PE3 end-to-end test is added (+37), and USES_PYTHON_SOURCE_TYPE = True is added to static_patterns_privilege_escalation.py. The conflict resolutions in static_runner.py, build_context.py and test_shared_python_ast.py keep both sides and look correct. Net PR diff against its merge base: 31 files (+11,690/-453) at 3f2f7f7, 32 files (+12,838/-471) now. Please land fixes as their own commits so reviewers can isolate them.

Material findings

1. [Blocker] src/skillspector/nodes/analyzers/static_python_shell_truthiness.py:1135, :1565 with static_patterns_tool_misuse.py:5483: TM1 detections that main reports are now lost. _reconcile_variable_shell_findings drops every same-scope x = True ... shell=x lexical finding (same_scope or ownership False) and leaves it to the companion. The companion now declines in common code:

   • _advance_trusted_names clears trust for every unsupported statement (line 1135). A trailing if __name__ == "__main__": therefore records an invalidation after every earlier def, and line 1519 scans those function bodies with no trust.
   • Once any generic call sets unknown_unsafe_bindings, every later import clears trust, including import subprocess (line 1565).

   Source-traced failing inputs. Main reports TM1 HIGH for both via _VARIABLE_SHELL_FLAG_RE. This head reports nothing:

   import subprocess
   
   def run(cmd):
       use_shell = True
       subprocess.run(cmd, shell=use_shell)
   
   if __name__ == "__main__":
       run("ls")

   import os
   payload = input()
   import subprocess
   enabled = True
   subprocess.run(output, shell=enabled)

   The second input is the original test_preparsed_python_is_reused_by_all_ast_analyzers fixture. #578's 68e22d0 reorders it to put the import first instead of keeping the detection. The same reconciliation also drops main's finding for return subprocess.run(cmd, shell=use_shell) because Return is never inspected. That part predates this head.
   Expected fix: when the companion cannot prove receiver identity or truthiness, keep main's lexical TM1 (reduced confidence is fine). Suppress it only when the receiver is explicitly rebound (store, import alias, attribute mutation) or the companion emits its own finding for that call. Fix this in the shared #577 module and propagate it through #578 and this PR. Add node()-level regressions for both inputs above.

2. [Non-blocking] The PR body is stale. It still cites 59a8d9f, #577 at 995d746, #578 at 1c03702 and the review range 1c03702..59a8d9f. Please update it to the current heads and the Python-specific range (68e22d0..a4d8525).

PIC tradeoffs: The receiver-trust model now treats any statement that could in theory rebind subprocess as if it did. That includes a later call, a compound statement, a class with any base, and a __del__ finalizer. This removes contrived proxy false positives but costs detections in ordinary scripts. The PIC should confirm that an uncertain receiver keeps the lexical TM1 finding rather than dropping it.

Verification and gaps: I read all four prior rng1995 reviews, both inline threads and the replies. I compared the PR diff against its merge base at 3f2f7f7 and at a4d8525 per file and inspected the remerge diff of a4d8525. I traced the forward scan, both prepasses, _reconcile_variable_shell_findings and main's _VARIABLE_SHELL_FLAG_RE / _variable_shell_flag_same_scope path for the inputs above. I sampled Python source classification and the build_context decode/ambiguous handling, which fails closed as PARTIAL (PYTHON_SOURCE_DECODE_ERROR / PYTHON_SOURCE_AMBIGUOUS), and found no issue there. This is not a full re-assessment of the 32-file stack. Env-shebang parsing, provider boundaries, nested/transitive analysis and reporting still need a complete pass once the blocker is fixed. All six checks pass. Per policy, tests and contributor code were not executed locally.

---

Decision: Changes Requested (reviewed head a4d8525e911127ddd86e8baa7336b59dd8b17524)

[rng1995]

Re-reviewed exact head 34772170b8b6e68d16b88bec42d4cb3b38a424dc against all earlier reviews and the full current diff. The reported original lexical-abstention cases now retain HIGH TM1, and focused analyzer/runner/end-to-end tests pass. Independent base-to-head reproductions confirm three P1 groups: untaken expression stores, identity-preserving receiver/flag stores, and an earlier compound-statement invocation hidden by a future module store. Each still executes the native subprocess call with shell=True but loses its HIGH finding on this head. Details and minimal reproductions are inline. The green hosted checks do not cover these regressions. Changes requested; do not merge until they are fixed. The three earlier threads whose exact examples are now fixed are resolved; #577 remains an open dependency. Two additional P2 issues concern missed Python-specific supply-chain coverage and correctly decoded primary source still retaining a fatal unsupported-content event. No real provider inference was performed; runtime execution probes used inert capture.

[rng1995]

[SkillSpector Review]

Hi @chrisknvidia, thank you for the execution-surface classification work and for the detailed end-to-end coverage across .py, .pyw, shebang and Markdown surfaces!

Value and readiness: Ready once CI passes on the new head. Every finding from the last round is fixed: the three TM1 P1s and both P2s. The branch now builds on the merged #577, and on #578's fixes, which are pushed to #578. Please merge #578 before this PR. This branch contains #578's changes, so merging #578 first keeps that PR's history and review in place.

Previous findings (review at 3477217)

• [P1] Untaken expression stores: Resolved in bbdf081, the same change as #577 and #578.
• [P1] Native aliases and truthy unpacking: Resolved in bbdf081.
• [P1] Future store hiding an earlier invocation: Resolved in bbdf081.
• [P2] Supply-chain analysis on Python source-type surfaces: Resolved in efecb7a. static_patterns_supply_chain now opts into USES_PYTHON_SOURCE_TYPE. The literal-XOR curl | sh helper now gets SC2 HIGH as runner and runner.md too, not only .py/.pyw. Graph-level parity tests cover all four.
• [P2] PEP 263 primary Python rejected before its decode: Resolved in 4b0043f. A selected Python primary that decodes with its declared encoding no longer hits the generic UTF-8 rejection. Direct scans of latin-1 runner.py, runner.pyw and an extensionless shebang file now complete at 100%. An unknown codec, or bytes invalid for the declared codec, still fail fatally. The waiver respects the shared deadline.

What I changed on the branch (all signed off)

• eb74b76, bbdf081 and 8ed05b6: the same main merge, TM1 fix and marker-ownership test adaptation as #578.
• 2ba149d and 7ef4bf1: #578's duplicate and true-prefixed detection fixes, cherry-picked. A differential over 2,400 inputs finds no call that #577 reports and this branch misses, and no new duplicates.
• efecb7a and 4b0043f: the two P2 fixes above.
• a4131c9: merges current main (with #577), keeping this branch's superset version in the four files #577 also changed.

Audit note (no change needed): anti_refusal and prompt_injection also branch on file_type == "python", but they are deliberately not opted in. For them, opting in could only drop AR2 false positives or drop P2 coverage, so leaving them out never loses a detection.

Verification:

• Full local non-integration suite on a4131c9: 11,785 passed, 0 failed.
• The TM1 probe matches #577 and #578.
• ruff is clean.

---

Decision: Ready to merge once CI passes, after #578. A human maintainer needs to approve (reviewed head a4131c9c8c3518898595268bc07a99c05588a6e6). This bot pushed commits to the branch, so it does not approve the PR itself.

[rng1995]

[SkillSpector Review]

@chrisknvidia, a short follow-up to the review above. 8d4b7d9 records #578's reviewed head (4031711) as an ancestor of this branch. The tree is byte-identical to a4131c9, which I reviewed and fully tested (11,785 passed), so nothing in the code changed.

Merge order for maintainers: merge #578 first, using "Create a merge commit", not squash. This PR then merges current main without conflicts, and its CI on 8d4b7d9 covers the resulting tree. A squash merge of #578 would conflict with this branch in four shared files and need another round of conflict resolution and CI.

---

Decision: Ready to merge once CI passes, after #578. A human maintainer needs to approve (reviewed head 8d4b7d9500200cac5ac9b763b2833481c1b10c90). This bot pushed commits to the branch, so it does not approve the PR itself.

[rng1995]

[SkillSpector Review]

@chrisknvidia, #578 landed on main as squash commit dd95368. Its tree is identical to #578's reviewed head, which is already an ancestor of this branch. e89f08d merges main and keeps this branch's version in the four overlapping files. The resulting tree is byte-identical to 8d4b7d9. That head was fully reviewed and tested (11,785 passed locally) and had all six CI checks green, so nothing in the code changed. The PR diff now shows only the execution-surface work (32 files).

---

Decision: Ready to merge once CI passes on the new head. A human maintainer needs to approve (reviewed head e89f08d11aa4feaa51cd10045b37c3fed7ea083f). This bot pushed commits to the branch, so it does not approve the PR itself.