Skip to content

XenonAnalyse: optional switch-aware block walker for forward-jumping switch tables#185

Open
Uproared wants to merge 3 commits intohedge-dev:mainfrom
Uproared:switch-aware-walker
Open

XenonAnalyse: optional switch-aware block walker for forward-jumping switch tables#185
Uproared wants to merge 3 commits intohedge-dev:mainfrom
Uproared:switch-aware-walker

Conversation

@Uproared
Copy link
Copy Markdown

Problem

Function::Analyze's block walker terminates on unconditional bctr
without pushing any successor blocks (function.cpp:149, 190-217).
The end-of-Analyze discontinuity sort-and-erase (function.cpp:237+)
then drops every block past the first address gap. For functions
whose primary control flow is switch-dispatched, this shrinks
fn.size to end roughly at the bctr, and every case label beyond
that point falls outside [fn.base, fn.base + fn.size).

Downstream consumers that do a containment check on switch labels
against this fn.size — notably XenonRecomp's recompiler
(XenonRecomp/recompiler.cpp:633) — then emit "Switch case trying to
jump outside function" warnings and replace the would-be
goto loc_<label>; in the generated C++ with // ERROR + return;.

The switch labels are, in fact, reachable (the bctr's computed
target at runtime is one of them). They are only "outside function"
from the block walker's perspective, because the walker didn't know
which targets the bctr could reach and conservatively pruned them.

Fix shape

Teach Function::Analyze about switch dispatches via a new optional
parameter:

static Function Analyze(const void* code, size_t size, size_t base,
                        const AnalyzerSwitchTableMap* switchMap = nullptr);

When the caller hands in a populated map keyed by BCTR guest VA, the
walker's unconditional-bctr path pushes every label (and the
default) as a successor block via the existing SearchBlock-guarded
emplace_back path. The end-of-Analyze discontinuity erase is
skipped when any route-A push fired (tracked via a local
switchAwareTermination flag), because switch dispatch leaves a
legitimate address gap between `bctr+4` (end of pre-switch block)
and the first case label — the jump-table bytes themselves live in
that gap, and the discontinuity heuristic would erase exactly the
blocks we just worked to make reachable.

Null-map safety is strict: when `switchMap == nullptr` (the
default), the switch-aware branch is skipped entirely and the walker
uses the legacy code path byte-for-byte. Every existing caller gets
identical behavior without any source change.

Commits

  1. `XenonAnalyse: add optional AnalyzerSwitchTableMap parameter to Function::Analyze` — signature only, no behavior change.
  2. `XenonAnalyse: block walker uses switchMap to push successors at known bctrs` — walker change.
  3. `XenonAnalyse: add synthetic PPC fixtures for switch-aware walker` — three hand-crafted tests (happy path, null-map safety, wrong-map miss).

Tests

`XenonAnalyse/tests/switch_aware_walker_test.cpp` exercises:

  • Happy path: switchMap populated, walker extends fn.size through the default label block.
  • Null-map safety: switchMap == nullptr, legacy truncation preserved.
  • Wrong-map miss: switchMap has an entry for an unrelated bctr VA, walker falls through to legacy.

All three assert on fn.size landing in the expected range. The tests are self-contained synthetic bytecode; no proprietary test corpus required.

```
[ok happy-path] fn.base=0x10000, fn.size=0x38, blocks=7
[ok null-map ] fn.base=0x10000, fn.size=0x1C (legacy walker truncation preserved)
[ok wrong-map] fn.base=0x10000, fn.size=0x1C (unrelated map entry correctly ignored)
All 3 tests PASSED
```

CMake integration for the tests is NOT included — happy to add an opt-in `-DXENONANALYSE_TESTS=ON` target or merge into an existing test harness if one is preferred.

Key design choice

The map is keyed by BCTR guest VA, not by XenonAnalyse's own `SwitchTable::base` (first-instruction-of-pattern VA). The caller does the switch-base → bctr-VA offset math at map-construction time (including handling any NOP padding between MTCTR and BCTR that some compilers emit for alignment), and hands the walker a pre-keyed map. This decouples the walker from pattern-length / pattern-type knowledge and keeps the walker simple.

Observed impact

A downstream recompile project (Xbox 360 PPC binary; decades-old compiler output) was emitting 2,304 "Switch case trying to jump outside function" warning lines across 76 unique bctr sites before this change. After wiring the optional switchMap into the recompiler's Function::Analyze call sites, 74 of 76 sites resolve cleanly (97.4%); the two residuals are unrelated pre-existing edge cases in switch-table ScanTable heuristics and are not route-A regressions.

Happy to adjust naming, split differently, fold tests into an existing runner, or any other shape you'd prefer.

…on::Analyze

Defines AnalyzerSwitchTable { defaultLabel, labels[] } and
AnalyzerSwitchTableMap = unordered_map<uint32_t /* bctr VA */,
AnalyzerSwitchTable>. Function::Analyze gains an optional
`const AnalyzerSwitchTableMap* switchMap = nullptr` parameter.

Default nullptr preserves existing behavior exactly: no call site
needs to change, and the walker currently ignores the parameter.

The next commit wires the parameter into the walker's unconditional-
bctr handling: when switchMap is populated and the walker reaches a
bctr whose guest VA has a map entry, it will push every switch label
(and the default) as a successor block rather than terminating.

Rationale for keying by BCTR VA rather than by XenonAnalyse's own
`SwitchTable::base` (first-insn-of-pattern VA): the producer of the
TOML-emitted switch tables and the consumer of Function::Analyze are
often the same caller, but the walker encounters instructions at the
BCTR's address directly. Keying the map by BCTR VA removes the need
for the walker to track pattern offsets (6 insns for absolute-form
switches, 8 for computed / short-offset, 7 for byte-offset) or to
handle NOP padding variations — the caller does the offset math once
at map-construction time.

Fixes the well-known class of issues where switch-dispatched functions
have their successor blocks swept away by the discontinuity pass, and
downstream consumers (e.g., recompilers) emit "jump outside function"
errors for labels that are actually reachable. Observed on an Xbox 360
title whose compiler emitted forward-jumping switch tables in this
shape, and referenced in UnleashedRecomp-lineage work; other Xbox 360
recompile targets likely hit it too.

See also:
- A follow-up test commit adds synthetic hand-crafted PPC fixtures
  covering the happy path + null-map safety + wrong-map miss.
… bctrs

When Function::Analyze's block walker reaches an unconditional bctr
(PPC_OP_CTR, xop 528, BO bit 5 set) and switchMap is non-null and
contains an entry for the bctr's guest VA, the walker pushes every
label (and the default) as a successor block via the existing
SearchBlock-guarded emplace path. The walker then reaches those
blocks, their reachability propagates, and `fn.size` correctly
includes the switch-dispatched code.

Two guards match the existing `branchDest < base` guard for
unconditional `b` (tail-call pattern):

  label < base           → skip. Labels before the function base are
                           malformed or tail-call-style jumps; they
                           should not extend fn.size.
  label >= base + size   → skip. Labels outside the caller's window
                           cannot be walked safely; emplacing them
                           would compute an out-of-buffer data pointer
                           during RESTORE_DATA.

Discontinuity-pass suppression: when the walker pushes switch-label
successors (tracked via local `switchAwareTermination`), the end-of-
Analyze sort-and-erase pass skips the erase. Switch dispatch leaves
a legitimate address gap between bctr+4 (end of pre-switch block)
and the first case label — the jump-table bytes live in that gap,
and the generic discontinuity heuristic would erase every block past
it, including exactly the label blocks we just pushed.

The sort itself still runs, which is fine: fn.size is computed as
max(block.base + block.size) across all blocks afterward, so order
does not change the result.

Null-map safety: when switchMap is nullptr, the switch-aware branch
is skipped entirely; the walker's unconditional-bctr path falls
through to the legacy "pop without successors" behavior. Output is
byte-identical to pre-patch behavior for all callers that pass
nullptr (the default) at the Analyze call site.

A companion test commit adds synthetic hand-crafted PPC bytecode
fixtures covering the happy path + null-map safety + wrong-map miss.
Three test scenarios hand-crafted in-source from big-endian PPC
instruction words, no proprietary bytecode. Each exercises a
different path through Function::Analyze's new switchMap handling:

  1. Happy path: switchMap populated with an entry for the synthetic
     function's bctr. Walker pushes all 4 case labels + default as
     successor blocks; fn.size covers through the default block.

  2. Null-map safety: switchMap = nullptr. Walker uses legacy pre-
     patch behavior; discontinuity pass erases label blocks; fn.size
     covers only the pre-bctr head. Guards against the default-null
     behavior regressing.

  3. Wrong-map miss: switchMap populated but with an entry for an
     unrelated bctr VA. switchMap->find(our_bctr) misses; walker
     falls through to legacy. Guards against the switchMap handling
     spuriously firing on wrong entries.

Assertions use fprintf + return-1 rather than a test framework, to
keep the test self-contained. If the repository later adopts a
standard test runner (Catch2 / doctest / gtest), porting is
straightforward.

See tests/README.md for the manual build command until CMake
integration is decided.
@Uproared Uproared force-pushed the switch-aware-walker branch from 3db42bf to fce79a7 Compare April 24, 2026 09:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant