XenonAnalyse: optional switch-aware block walker for forward-jumping switch tables#185
Open
Uproared wants to merge 3 commits intohedge-dev:mainfrom
Open
XenonAnalyse: optional switch-aware block walker for forward-jumping switch tables#185Uproared wants to merge 3 commits intohedge-dev:mainfrom
Uproared wants to merge 3 commits intohedge-dev:mainfrom
Conversation
…on::Analyze
Defines AnalyzerSwitchTable { defaultLabel, labels[] } and
AnalyzerSwitchTableMap = unordered_map<uint32_t /* bctr VA */,
AnalyzerSwitchTable>. Function::Analyze gains an optional
`const AnalyzerSwitchTableMap* switchMap = nullptr` parameter.
Default nullptr preserves existing behavior exactly: no call site
needs to change, and the walker currently ignores the parameter.
The next commit wires the parameter into the walker's unconditional-
bctr handling: when switchMap is populated and the walker reaches a
bctr whose guest VA has a map entry, it will push every switch label
(and the default) as a successor block rather than terminating.
Rationale for keying by BCTR VA rather than by XenonAnalyse's own
`SwitchTable::base` (first-insn-of-pattern VA): the producer of the
TOML-emitted switch tables and the consumer of Function::Analyze are
often the same caller, but the walker encounters instructions at the
BCTR's address directly. Keying the map by BCTR VA removes the need
for the walker to track pattern offsets (6 insns for absolute-form
switches, 8 for computed / short-offset, 7 for byte-offset) or to
handle NOP padding variations — the caller does the offset math once
at map-construction time.
Fixes the well-known class of issues where switch-dispatched functions
have their successor blocks swept away by the discontinuity pass, and
downstream consumers (e.g., recompilers) emit "jump outside function"
errors for labels that are actually reachable. Observed on an Xbox 360
title whose compiler emitted forward-jumping switch tables in this
shape, and referenced in UnleashedRecomp-lineage work; other Xbox 360
recompile targets likely hit it too.
See also:
- A follow-up test commit adds synthetic hand-crafted PPC fixtures
covering the happy path + null-map safety + wrong-map miss.
… bctrs
When Function::Analyze's block walker reaches an unconditional bctr
(PPC_OP_CTR, xop 528, BO bit 5 set) and switchMap is non-null and
contains an entry for the bctr's guest VA, the walker pushes every
label (and the default) as a successor block via the existing
SearchBlock-guarded emplace path. The walker then reaches those
blocks, their reachability propagates, and `fn.size` correctly
includes the switch-dispatched code.
Two guards match the existing `branchDest < base` guard for
unconditional `b` (tail-call pattern):
label < base → skip. Labels before the function base are
malformed or tail-call-style jumps; they
should not extend fn.size.
label >= base + size → skip. Labels outside the caller's window
cannot be walked safely; emplacing them
would compute an out-of-buffer data pointer
during RESTORE_DATA.
Discontinuity-pass suppression: when the walker pushes switch-label
successors (tracked via local `switchAwareTermination`), the end-of-
Analyze sort-and-erase pass skips the erase. Switch dispatch leaves
a legitimate address gap between bctr+4 (end of pre-switch block)
and the first case label — the jump-table bytes live in that gap,
and the generic discontinuity heuristic would erase every block past
it, including exactly the label blocks we just pushed.
The sort itself still runs, which is fine: fn.size is computed as
max(block.base + block.size) across all blocks afterward, so order
does not change the result.
Null-map safety: when switchMap is nullptr, the switch-aware branch
is skipped entirely; the walker's unconditional-bctr path falls
through to the legacy "pop without successors" behavior. Output is
byte-identical to pre-patch behavior for all callers that pass
nullptr (the default) at the Analyze call site.
A companion test commit adds synthetic hand-crafted PPC bytecode
fixtures covering the happy path + null-map safety + wrong-map miss.
Three test scenarios hand-crafted in-source from big-endian PPC
instruction words, no proprietary bytecode. Each exercises a
different path through Function::Analyze's new switchMap handling:
1. Happy path: switchMap populated with an entry for the synthetic
function's bctr. Walker pushes all 4 case labels + default as
successor blocks; fn.size covers through the default block.
2. Null-map safety: switchMap = nullptr. Walker uses legacy pre-
patch behavior; discontinuity pass erases label blocks; fn.size
covers only the pre-bctr head. Guards against the default-null
behavior regressing.
3. Wrong-map miss: switchMap populated but with an entry for an
unrelated bctr VA. switchMap->find(our_bctr) misses; walker
falls through to legacy. Guards against the switchMap handling
spuriously firing on wrong entries.
Assertions use fprintf + return-1 rather than a test framework, to
keep the test self-contained. If the repository later adopts a
standard test runner (Catch2 / doctest / gtest), porting is
straightforward.
See tests/README.md for the manual build command until CMake
integration is decided.
3db42bf to
fce79a7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Function::Analyze's block walker terminates on unconditionalbctrwithout pushing any successor blocks (
function.cpp:149,190-217).The end-of-Analyze discontinuity sort-and-erase (
function.cpp:237+)then drops every block past the first address gap. For functions
whose primary control flow is switch-dispatched, this shrinks
fn.sizeto end roughly at thebctr, and every case label beyondthat point falls outside
[fn.base, fn.base + fn.size).Downstream consumers that do a containment check on switch labels
against this
fn.size— notably XenonRecomp's recompiler(
XenonRecomp/recompiler.cpp:633) — then emit "Switch case trying tojump outside function" warnings and replace the would-be
goto loc_<label>;in the generated C++ with// ERROR+return;.The switch labels are, in fact, reachable (the
bctr's computedtarget at runtime is one of them). They are only "outside function"
from the block walker's perspective, because the walker didn't know
which targets the
bctrcould reach and conservatively pruned them.Fix shape
Teach
Function::Analyzeabout switch dispatches via a new optionalparameter:
When the caller hands in a populated map keyed by BCTR guest VA, the
walker's unconditional-bctr path pushes every label (and the
default) as a successor block via the existing
SearchBlock-guardedemplace_backpath. The end-of-Analyze discontinuity erase isskipped when any route-A push fired (tracked via a local
switchAwareTerminationflag), because switch dispatch leaves alegitimate address gap between `bctr+4` (end of pre-switch block)
and the first case label — the jump-table bytes themselves live in
that gap, and the discontinuity heuristic would erase exactly the
blocks we just worked to make reachable.
Null-map safety is strict: when `switchMap == nullptr` (the
default), the switch-aware branch is skipped entirely and the walker
uses the legacy code path byte-for-byte. Every existing caller gets
identical behavior without any source change.
Commits
Tests
`XenonAnalyse/tests/switch_aware_walker_test.cpp` exercises:
All three assert on fn.size landing in the expected range. The tests are self-contained synthetic bytecode; no proprietary test corpus required.
```
[ok happy-path] fn.base=0x10000, fn.size=0x38, blocks=7
[ok null-map ] fn.base=0x10000, fn.size=0x1C (legacy walker truncation preserved)
[ok wrong-map] fn.base=0x10000, fn.size=0x1C (unrelated map entry correctly ignored)
All 3 tests PASSED
```
CMake integration for the tests is NOT included — happy to add an opt-in `-DXENONANALYSE_TESTS=ON` target or merge into an existing test harness if one is preferred.
Key design choice
The map is keyed by BCTR guest VA, not by XenonAnalyse's own `SwitchTable::base` (first-instruction-of-pattern VA). The caller does the switch-base → bctr-VA offset math at map-construction time (including handling any NOP padding between MTCTR and BCTR that some compilers emit for alignment), and hands the walker a pre-keyed map. This decouples the walker from pattern-length / pattern-type knowledge and keeps the walker simple.
Observed impact
A downstream recompile project (Xbox 360 PPC binary; decades-old compiler output) was emitting 2,304 "Switch case trying to jump outside function" warning lines across 76 unique bctr sites before this change. After wiring the optional switchMap into the recompiler's Function::Analyze call sites, 74 of 76 sites resolve cleanly (97.4%); the two residuals are unrelated pre-existing edge cases in switch-table ScanTable heuristics and are not route-A regressions.
Happy to adjust naming, split differently, fold tests into an existing runner, or any other shape you'd prefer.