Skip to content

Commit 7ff83e6

Browse files
committed
oracle: confirm the u64 rotate refusal is the OPERATION, not the idiom
Rung 6 of the ladder rests on one measurement: a scalar u64 rotate loop does not vectorize. That used a single source form -- u64::rotate_right(n) -- which leaves an obvious objection open. Maybe LLVM declines the rotate IDIOM at 64-bit width and an explicit shift-or would vectorize, exactly as it does for u32's rotate_left(12)/(7). AVX2 has vpsllq/vpsrlq, so the ingredients exist. Tested. It does not. rot_u64x8 u64::rotate_right(n) runtime 0 packed shiftor_rot_u64x8 (x >> n) | (x << (64-n)) runtime 0 packed shiftor_rot_const_u64x8 explicit shift-or CONST 0 packed The third row is the sharp one. At u32 width a constant amount vectorizes two different ways depending on granularity -- byte-granular to vpshufb, bit-granular to the shift-or triple -- and both are packed. At u64 width neither happens, for either kind of constant. A fourth confirmation arrived unasked. A blake2b_g_shiftor_u64x8 probe was written as the direct counterpart to blake2b_g_u64x8; it never appeared in the emitted assembly. LLVM FOLDED the two functions -- zero mentions of the shiftor symbol, two call sites on the survivor -- proving the spellings are byte-identical rather than merely both-scalar. Removed rather than kept as a permanently-failing row: a probe the compiler cannot distinguish from its own control measures nothing. The removal and its reason are recorded in the probe file and the baseline so the next reader does not re-add it. Net: rung 6 is confirmed as the crate's one earned intrinsic override, now on three independent spellings instead of one.
1 parent 712a6a6 commit 7ff83e6

3 files changed

Lines changed: 121 additions & 0 deletions

File tree

.claude/knowledge/crypto-lane-status.md

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,35 @@ Two things make this a genuine finding rather than a shrug:
8686
and rotate-by-7. It has the tools and declines to apply them at 64-bit
8787
width.
8888

89+
### Confirmed a second and third way: it is the OPERATION, not the idiom
90+
91+
The measurement above used one source form — `u64::rotate_right(n)`. That
92+
leaves an obvious objection: maybe LLVM declines the *rotate idiom* at 64-bit
93+
width, and an explicit shift-or would vectorize, exactly as it does for u32's
94+
`rotate_left(12)`/`(7)`. AVX2 has `vpsllq`/`vpsrlq`, so the ingredients exist.
95+
96+
Tested. It does not.
97+
98+
| probe | spelling | amount | packed |
99+
|---|---|---|---|
100+
| `rot_u64x8` | `u64::rotate_right(n)` | runtime | **0** |
101+
| `shiftor_rot_u64x8` | `(x >> n) \| (x << (64-n))` | runtime | **0** |
102+
| `shiftor_rot_const_u64x8` | explicit shift-or | **const** 32/24/16/63 | **0** |
103+
104+
The third row is the sharp one. At u32 width a *constant* amount vectorizes
105+
two different ways depending on granularity — byte-granular folds to
106+
`vpshufb`, bit-granular to the shift-or triple — and both are packed. At u64
107+
width neither happens, for either kind of constant.
108+
109+
**And a fourth confirmation arrived unasked.** A `blake2b_g_shiftor_u64x8`
110+
probe was written as the direct counterpart to `blake2b_g_u64x8`. It never
111+
appeared in the emitted assembly: LLVM **folded the two functions**, leaving
112+
zero mentions of the shiftor symbol and two call sites pointing at the
113+
survivor. So the two spellings are not merely both-scalar — they are
114+
*byte-identical*. The probe was removed rather than kept as a permanently
115+
failing row, since a probe the compiler cannot distinguish from its own
116+
control measures nothing.
117+
89118
**So the u64 ARX lane is the crate's first intrinsic override that meets the
90119
entry criterion** (a probe proving the generic form fails). AVX-512:
91120
`_mm512_rorv_epi64` / `VPROLVQ`, one instruction. AVX2 / NEON / wasm: write

.claude/knowledge/simd-codegen-oracle/baseline-x86_64-v3.toml

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -168,3 +168,25 @@ note = "MEASURED FULLY PACKED, ZERO SPILL. Observed 98 packed / 0 scalar-lane-ar
168168
[probe.arx_lane512_x8]
169169
expect = "unknown"
170170
note = "MEASURED FULLY PACKED. Observed 81 packed / 0 scalar-lane-arith / 0 loop-control / 0 memory: 16 vpaddd + 16 vpxor + 16 vpshufb + 33 vmovdqa. Bit-for-bit identical output to arx_node4096 (asserted in the driver), and identical arithmetic in the emitted code -- so the node-wide unit costs nothing structurally over eight applications of the 512-bit lane."
171+
172+
# Group C, continued -- the u64 rotate written as an EXPLICIT shift-or rather
173+
# than as `u64::rotate_right`. Group C above tested one SOURCE FORM; this tests
174+
# whether LLVM declines the rotate IDIOM at 64-bit width or declines the
175+
# OPERATION. Decides whether ladder rung 6 needs an intrinsic override.
176+
#
177+
# ANSWER: it declines the OPERATION. Both spellings come back 0 packed, for
178+
# runtime-variable AND compile-time-constant amounts. A third probe
179+
# (blake2b_g_shiftor_u64x8) was written and removed: LLVM FOLDED it into
180+
# blake2b_g_u64x8 -- zero mentions of the shiftor symbol in the emitted asm,
181+
# two call sites on the survivor -- proving the two spellings are byte-identical
182+
# rather than merely both-scalar. A probe the compiler cannot distinguish from
183+
# its own control measures nothing.
184+
185+
[probe.shiftor_rot_u64x8]
186+
expect = "unknown"
187+
note = "MEASURED SCALAR -- 0 packed / 8 scalar-lane-arith / 19 memory, i.e. indistinguishable from rot_u64x8. Writing the rotate as an explicit (x >> n) | (x << (64-n)) does NOT persuade LLVM to vectorize it, so the refusal is about the 64-bit OPERATION and not about the rotate idiom. Driver asserts equality with u64::rotate_right, so this is the same function in a different spelling."
188+
189+
[probe.shiftor_rot_const_u64x8]
190+
expect = "unknown"
191+
note = "MEASURED SCALAR -- 0 packed / 8 scalar-lane-arith / 8 memory, with BLAKE2b's COMPILE-TIME-CONSTANT 32/24/16/63. This is the sharper half: at u32 width a constant byte-granular amount folds to vpshufb and a constant bit-granular one folds to shift-or, both packed. At u64 width neither happens, for either kind of constant. A third independent confirmation that the u64 lane is the crate's one earned intrinsic override."
192+

.claude/knowledge/simd-codegen-oracle/probes.rs

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -493,6 +493,51 @@ pub fn transpose_16x16_composed(m: [U32x16; 16]) -> [U32x16; 16] {
493493
m
494494
}
495495

496+
/// The u64 rotate written as an EXPLICIT shift-or, not as `u64::rotate_right`.
497+
///
498+
/// Group C measured that `u64::rotate_right(n)` lowers to a scalar `rorq` per
499+
/// lane — 0 packed. But that tested one SOURCE FORM. LLVM already vectorizes
500+
/// u32's `rotate_left(12)`/`(7)` as `vpslld`+`vpsrld`+`vpor`, and AVX2 has the
501+
/// 64-bit equivalents `vpsllq`/`vpsrlq`. So the open question is whether LLVM
502+
/// declines the *rotate idiom* at 64-bit width, or declines the *operation*.
503+
///
504+
/// If the explicit form vectorizes, rung 6 of the ladder needs no intrinsic
505+
/// override at all — just a differently-spelled body, exactly as rung 2 needed
506+
/// only index loops. If it does not, rung 6 is the crate's first genuinely
507+
/// earned `unsafe` intrinsic.
508+
///
509+
/// `n` is constrained to `1..=63` by the caller: `x >> 64` is UB on `u64`, so
510+
/// the zero case must be excluded rather than masked, matching how
511+
/// `simd_nightly`'s `U32x16::rotate_left` guards `n % 32 == 0`.
512+
#[inline(always)]
513+
fn shiftor_rotr_u64x8(v: U64x8, n: u32) -> U64x8 {
514+
let a = v.to_array();
515+
let mut out = [0u64; 8];
516+
for i in 0..8 {
517+
out[i] = (a[i] >> n) | (a[i] << (64 - n));
518+
}
519+
U64x8::from_array(out)
520+
}
521+
522+
/// Explicit shift-or u64 rotate, runtime-variable amount, 8 lanes.
523+
#[inline(never)]
524+
pub fn shiftor_rot_u64x8(v: U64x8, n: u32) -> U64x8 {
525+
shiftor_rotr_u64x8(v, n)
526+
}
527+
528+
/// Explicit shift-or u64 rotate with COMPILE-TIME-CONSTANT amounts — BLAKE2b's
529+
/// 32/24/16/63. Separated from the runtime-variable probe above because the
530+
/// u32 lane vectorizes constants and variables differently (`vpshufb` for
531+
/// byte-granular constants, shift-or otherwise), so the two cases must be
532+
/// distinguished rather than averaged.
533+
#[inline(never)]
534+
pub fn shiftor_rot_const_u64x8(v: U64x8) -> U64x8 {
535+
let a = shiftor_rotr_u64x8(v, 32);
536+
let b = shiftor_rotr_u64x8(a, 24);
537+
let c = shiftor_rotr_u64x8(b, 16);
538+
shiftor_rotr_u64x8(c, 63)
539+
}
540+
496541
// ============================================================================
497542
// Group E — UNKNOWN. Is 4096 bit / 512 byte a viable DEFAULT lane?
498543
// ============================================================================
@@ -651,6 +696,31 @@ fn main() {
651696
);
652697
acc ^= ga.reduce_sum() ^ gb.reduce_sum() ^ gc.reduce_sum() ^ gd.reduce_sum();
653698

699+
// ---- Group C, explicit shift-or form ----
700+
let sv = U64x8::from_array(std::array::from_fn(|_| rng.next()));
701+
let sn = 1 + (rng.next() % 63) as u32;
702+
// The explicit form must agree with `u64::rotate_right` -- same function,
703+
// different spelling. A codegen probe that measured a DIFFERENT function
704+
// would be worthless.
705+
let want: [u64; 8] = {
706+
let a = sv.to_array();
707+
std::array::from_fn(|i| a[i].rotate_right(sn))
708+
};
709+
let got = shiftor_rot_u64x8(black_box(sv), black_box(sn));
710+
assert_eq!(got.to_array(), want, "shift-or rotate != u64::rotate_right");
711+
acc ^= got.reduce_sum();
712+
713+
acc ^= shiftor_rot_const_u64x8(black_box(sv)).reduce_sum();
714+
715+
// NOTE: a `blake2b_g_shiftor_u64x8` probe was written here and REMOVED,
716+
// because LLVM folded it into `blake2b_g_u64x8` -- identical machine code,
717+
// zero mentions of the shiftor symbol in the emitted asm, and two call
718+
// sites pointing at the surviving one. That fold is the result: the
719+
// explicit shift-or spelling and `u64::rotate_right` are not merely both
720+
// scalar, they are BYTE-IDENTICAL. A probe the compiler cannot tell apart
721+
// from its own control measures nothing, so it is gone rather than kept
722+
// as a permanently-failing row.
723+
654724
// ---- Group D ----
655725
let mut rand_u32x16 = || U32x16::from_array(std::array::from_fn(|_| rng.next() as u32));
656726
let (b3a, b3b, b3c, b3d) = blake3_g_u32x16(

0 commit comments

Comments
 (0)