UI-over-3D compositing artifacts: menu dim renders as triangles + stale back-buffer ghosting (D3DSWAPEFFECT_DISCARD)

[fadi-labib]

Summary

Two related UI-over-3D compositing artifacts appear when translucent menus/dialogs are layered over the in-game scene. Not texture-related — reproduces with and without the RGBA8 texture overlay (issue #9), and the pure-2D Skirmish lobby renders perfectly. Seen on S22 Ultra (Xclipse 920, DXVK), but the causes are driver-agnostic and likely reproduce on Adreno too.

Symptom A — in-game menu dim renders as hard triangles

Opening the ESC in-game menu should dim the scene behind it with a smooth translucent black rectangle. Instead the dim shows as large diagonal triangular wedges (an X/diamond across the screen).

• Path: the dim/panels are drawn via W3DDisplay::drawFillRect → m_2DRender->Add_Rect(...) under setup2DRenderState(nullptr, DRAW_IMAGE_ALPHA, FALSE) (W3DDisplay.cpp:2535).
• A full-screen filled rect = two triangles; seeing only a triangle (or mismatched halves) points to backface-cull / vertex-winding or per-vertex-alpha handling of the 2D quad under DXVK. Suspect the cull mode or vertex order in Render2DClass::Add_Rect/Render.

Symptom B — stacked dialogs ghost the previous frame

Opening the "Exit?" confirmation over the in-game menu shows the underlying menu and stale UI bleeding through the dialog (double "COMMAND & CONQUER", rows of dark blobs, ghosted button labels).

• Root cause is almost certainly the swap effect (dx8wrapper.cpp:1320):

  _PresentParameters.SwapEffect = D3DSWAPEFFECT_DISCARD; // "Shouldn't this be D3DSWAPEFFECT_FLIP?"
  _PresentParameters.BackBufferCount = IsWindowed ? 1 : 2;

  With DISCARD, back-buffer contents are undefined after Present. The 3D scene is fine because it clears+redraws every frame, but when a menu pauses the world the game redraws only the UI over whatever is in the back buffer — under DXVK's flip/MAILBOX presentation (log shows Present mode: VK_PRESENT_MODE_MAILBOX_KHR, Image count: 5) that's several-frames-old content, so old UI frames accumulate/bleed through. The original code even flags this with the // Shouldn't this be FLIP? comment.

Fix directions

• B: ensure a full color-buffer clear every frame even while paused in a menu, or configure a present/swap path that preserves back-buffer contents for the UI-only case. Cheapest: force a clear in the paused/menu render path.
• A: audit cull mode / vertex winding / alpha in the Render2DClass quad path so both triangles of a drawFillRect render consistently under DXVK.

Not this bug

Depth-buffer z-fighting (D3D9Format::D16) is a separate, unrelated artifact class.

Repro

Start a skirmish → press ESC (Symptom A) → click EXIT GAME (Symptom B). Screenshots available (menu-dim wedges; exit-dialog ghosting; clean lobby for contrast).

Environment

S22 Ultra (Exynos 2200 / Xclipse 920), stock Vulkan 1.3, DXVK 2.6, main @ current.

[fadi-labib]

Root cause found — single bug, correcting the earlier analysis

Correction: the initial "Symptom B = stale back-buffer / D3DSWAPEFFECT_DISCARD" theory is wrong. The in-game render loop calls WW3D::Begin_Render(true, true, ...) — clears color+depth — every frame (W3DDisplay.cpp:2134), so there is no back-buffer preservation to blame. Both symptoms are one bug.

Actual cause: 2D UI quads are backface-culled under DXVK

Render2DClass::Get_Default_Shader() (render2d.cpp:100) sets depth/blend/fog/gradient/texturing but never sets a cull mode, so it inherits the preset default CULL_MODE_ENABLE. The quad winding in Internal_Add_Quad_Indicies (render2d.cpp:378) is chosen for D3D8's front-face convention; on DXVK→Vulkan that convention flips, so one triangle of each 2D quad gets culled.

Because fills use Add_Quad and images use Add_Quad_Backfaced (opposite windings), one category culls while the other survives — which is exactly what the screenshots show:

• Symptom A: menu dim / panel fills render as diagonal half-triangles (wedges).
• Symptom B: dialog background fills are culled → the dialog is see-through and the menu behind "ghosts" through. Not stale-buffer at all.
• Button images render fine (their winding happens to survive), which is why the main menu and Skirmish lobby looked correct.

Fix (8d53e7dac, pending on-device confirmation)

2D UI must never backface-cull. One line in Get_Default_Shader:

shader.Set_Cull_Mode( ShaderClass::CULL_MODE_DISABLE );

Winding-independent → both triangles of every quad always render → fixes both symptoms and can't regress the elements that already worked.

Depth-buffer z-fighting (D16) remains a separate, unrelated item.