Re-import mlkem-native and mldsa-native; drop custom meta headers - #3367
Re-import mlkem-native and mldsa-native; drop custom meta headers#3367hanno-becker wants to merge 1 commit into
Conversation
4a4b530 to
16c4085
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3367 +/- ##
==========================================
+ Coverage 78.41% 78.47% +0.05%
==========================================
Files 695 696 +1
Lines 124223 124348 +125
Branches 17267 17303 +36
==========================================
+ Hits 97413 97584 +171
+ Misses 25886 25840 -46
Partials 924 924 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Reviewed, this covers the same ML-DSA x86 re-import and |
Update the pinned AWS-LC integration test from v5.1.0 to v5.4.0. The importer/glue in released AWS-LC predates the current mldsa-native API (dropped sig_len, renamed capability enums to MLD_SYS_CAP_X86_64_AVX2 and MLD_SYS_CAP_AARCH64_NEON). aws/aws-lc#3367 is the upstream re-import that makes AWS-LC compatible again, but it is not yet in any release, so its integration-layer changes are applied here as a pre-import patch: update importer.sh, ml_dsa.c, mldsa_native_config.h and mldsa_native_backend.h, and drop the custom mldsa_x86_64_meta.h / mldsa_aarch64_meta.h in favour of the vendored meta.h imported verbatim. This fixes our CI integration tests against AWS-LC main. Also exclude patch files from scripts/autogen's macro-typo check, since they carry old macro names on their context and deletion lines. - Fixes #1300 Signed-off-by: Matthias J. Kannwischer <matthias@zerorisc.com>
Update the pinned AWS-LC integration test from v5.1.0 to v5.4.0. The importer/glue in released AWS-LC predates the current mldsa-native API (dropped sig_len, renamed capability enums to MLD_SYS_CAP_X86_64_AVX2 and MLD_SYS_CAP_AARCH64_NEON). aws/aws-lc#3367 is the upstream re-import that makes AWS-LC compatible again, but it is not yet in any release, so its integration-layer changes are applied here as a pre-import patch: update importer.sh, ml_dsa.c, mldsa_native_config.h and mldsa_native_backend.h, and drop the custom mldsa_x86_64_meta.h / mldsa_aarch64_meta.h in favour of the vendored meta.h imported verbatim. This fixes our CI integration tests against AWS-LC main. Also exclude patch files from scripts/autogen's macro-typo check, since they carry old macro names on their context and deletion lines. - Fixes #1300 Signed-off-by: Matthias J. Kannwischer <matthias@zerorisc.com>
16c4085 to
8ce405c
Compare
ab497d4 to
8472767
Compare
|
CI legitimately failing because the code tries to use the PPC64LE backend which is not imported. |
8472767 to
3f3511d
Compare
3f3511d to
70590d9
Compare
|
mldsa-native v2 is imminent, so we should hold back merge until this PR is updated. |
Looks like v2 tasks are now complete, shall we do a fresh import here. I am happy otherwise. |
|
I think this PR needs to be updated to pull the v2.0.0 tags/commits of mldsa-native and mlkem-native? |
This commit re-imports mlkem-native (v2.0.0, d1b2fe782) and mldsa-native (v2.0.0, 834a90d5e), replacing mlkem-native v1.1.0 (d2cae2be) and mldsa-native 08d40f9403 as previously vendored. With the re-import, we remove the three custom backend meta headers previously owned and used by AWS-LC (ml_kem/mlkem_aarch64_meta.h, ml_dsa/mldsa_aarch64_meta.h, and ml_dsa/mldsa_x86_64_meta.h): - The AArch64 shadows existed to add a runtime NEON gate with a C fallback. Now, upstream gates NEON in the vendored meta.h itself. - The x86_64 mldsa shadow trimmed the backend to the assembly-backed subset. Upstream has since converted the remaining AVX2 C-intrinsic operations to proven assembly, so the full backend is imported and its meta.h is used as-is. The commit also removes a previous label-renaming at import time that was in place to avoid a name clash between mlkem and mldsa labels. Upstream has since made the labels globally unique (mlkem-native PR#1813 and mldsa-native PR#1310). The commit also removes a previous file-renaming at import time that was in place to avoid a name clash between the mlkem and mldsa assembly files, which are compiled into the same FIPS module. Upstream has since prefixed the backend assembly filenames per project (mlkem_*_asm.S and mldsa_*_asm.S), so the basenames are globally unique and the individual .S files can again be built directly. This commit also updates custom configurations to use the vendored capability enum while retaining AWS-LC mappings to CRYPTO_is_AVX2_capable() and CRYPTO_is_NEON_capable(). Both projects now qualify the enumerators by architecture, so mlkem_native_config.h uses MLK_SYS_CAP_X86_64_AVX2 and MLK_SYS_CAP_AARCH64_NEON, and mldsa_native_config.h the MLD_ equivalents. Without this, the build breaks outright, as the unqualified MLK_SYS_CAP_AVX2 / MLK_SYS_CAP_NEON no longer exist. The ML-KEM assembly source lists in crypto/fipsmodule/CMakeLists.txt now use file(GLOB ... CONFIGURE_DEPENDS), matching ML-DSA. This automatically absorbs upstream assembly-file additions, removals, and renames without corresponding CMake changes. Both importer scripts also allow GITHUB_SERVER_URL to be overridden for imports from local mirrors or forks. mlkem-native v2.0.0 is a breaking release. Beyond the capability enum above, it removes the deprecated MLK_CONFIG_API_* configuration mechanism and the SUPERCOP API (crypto_kem_*, CRYPTO_SECRETKEYBYTES, CRYPTO_BYTES, ...), both of which the vendored v1.1.0 still provided. AWS-LC configures mlkem-native through the recommended config file and never consumed the SUPERCOP names, so neither removal requires a change here; the latter does mean the FIPS module is no longer exposed to those generically named macros. Both projects also refine their error codes. mlkem-native replaces the single generic MLK_ERR_FAIL with MLK_ERR_INVALID_PK, MLK_ERR_INVALID_SK and MLK_ERR_PCT_FAIL for the FIPS 203 modulus check, hash check, and Pairwise Consistency Test respectively. mldsa-native adds MLD_ERR_SIGNING_PAUSED, MLD_ERR_INVALID_SIGNATURE, MLD_ERR_INVALID_KEY, MLD_ERR_PCT_FAIL and MLD_ERR_INVALID_ARG to the four codes previously vendored. The mldsa refinement is inconsequential here, because ml_dsa.c already normalised every mldsa-native return code to 1-on-success and never propagated the code itself. ml_kem.c had no such insulation: it returned mlkem-native's code verbatim, so the mlkem renumbering would silently have changed the values observable through ml_kem.h -- e.g. encapsulation against a malformed public key returning -4 rather than -1. See the return-value conventions below. Two further mldsa-native changes do not affect this integration: - MLD_CONFIG_NO_RANDOMIZED_API now gates keypair(), signature() and signature_extmu(), and MLD_CONFIG_CORE_API_ONLY is newly declared incompatible with MLD_CONFIG_KEYGEN_PCT. AWS-LC sets neither option. - Following the FIPS 204 errata @[FIPS204_UPDATES], the FIPS-compliant lower bound on MLD_CONFIG_MAX_SIGNING_ATTEMPTS rises from 814 to 821. AWS-LC does not set the option, so the fallback bound MLD_MAX_KAPPA / MLDSA_L applies -- 16382, 13106 and 9361 attempts for ML-DSA-44/65/87 -- comfortably above the new bound. API changes: Upstream mldsa-native has changed its sign and verify API to no longer accept a sig_len parameter. Instead, it assumes that the signature length is the static length associated with the chosen variant of ML-DSA. When moving from a higher-level API with variable-length signatures to mldsa-native's lower-level API, those length checks need to happen prior to calling into mldsa-native. The commit adds this in ml_dsa.c accordingly. The commit also fixes a minor API issue in the wrapper ml_dsa.c around mldsa-native whereby the nominally variable mu_len parameter was passed through to mldsa-native unchecked even though mldsa-native expects a buffer of static size 64 bytes. This was not exploitable since the next higher layer does check the length, rejecting message_len != pqdsa->digest_len in pkey_pqdsa_sign_generic() and pkey_pqdsa_verify_generic() (p_pqdsa.c:191 and :282). Still so long as ml_dsa.h exposes an API with variable-length mu, it should check for the expected length. The mldsa-native update also ensures that key generation and signing outputs are zeroized on failure. Return-value conventions: Neither ml_kem.h nor ml_dsa.h documented what their functions return, and the two use opposite conventions -- ml_dsa.h returns 1 on success per AWS-LC's usual convention (API-CONVENTIONS.md), while ml_kem.h inherited 0-on-success from mlkem-native. Both headers now state their convention and define named constants (ML_KEM_SUCCESS / ML_KEM_FAILURE, ML_DSA_SUCCESS / ML_DSA_FAILURE), each warning about the other. Each wrapper gains a static inline conversion function, ml_kem_result() and ml_dsa_result(), and every site that translates a vendored return code now goes through it. In ml_dsa.c this is behaviour-preserving, as the open-coded conversions it replaces already normalised to 1-on-success. In ml_kem.c it stops the pass-through described above: every failure now yields ML_KEM_FAILURE, so upstream's numbering is no longer part of AWS-LC's surface and future refinements are absorbed without further change. This does alter ML-KEM's failure values (the argument-rejection path in check_pk/check_sk previously returned -1, colliding with the value upstream now reserves for MLK_ERR_FAIL), but no caller is affected: crypto/fipsmodule/kem/kem.c, self_check.c, modulewrapper.cc and ml_kem_test.cc all test only for zero versus non-zero. The two files that call both APIs, self_check.c and modulewrapper.cc, must therefore use inverted success tests for the two modules. Both now carry a note to that effect at the relevant call sites, since an inverted test in a self-test or KAT would make the check pass unconditionally rather than fail. The PCT comments in ml_kem.c and ml_dsa.c now record why PCT failure is the only *reachable* keygen failure, so that AWS_LC_FIPS_failure() is not reached for an unrelated condition. All imported assembly has HOL Light functional-correctness and safety proofs upstream. All assembly except the rejection samplers additionally has proofs of secret-independent timing. Signed-off-by: Hanno Becker <beckphan@amazon.co.uk>
This re-imports mlkem-native and mldsa-native from their respective
mainbranches (mlkem-native 61c831345, mldsa-native 9a180a96b).Meta headers
This PR removes the three AWS-LC-owned "shadow" backend meta headers:
ml_kem/mlkem_aarch64_meta.h,ml_dsa/mldsa_aarch64_meta.h, andml_dsa/mldsa_x86_64_meta.h. The two AArch64 shadows (added in #3353)duplicated the vendored
meta.hto add a runtime NEON gate with C fallback (reachable viaOPENSSL_armcap=0); upstream now provides this gate itself (MLK_SYS_CAP_NEON/MLD_SYS_CAP_AARCH64_NEON), making theshadows exact duplicates. All backends consume the vendored
meta.has-is.Full x86_64 ML-DSA backend
The x86_64 mldsa shadow trimmed the vendored
meta.hto the assembly-backed subset that AWS-LC imported. The remaining AVX2 C-intrinsic operations have since been rewritten and proved in assembly upstream, so wenow import the full x86_64 backend.
Assembly local labels
Earlier revisions of this PR rewrote mldsa's local assembly labels to
Lmldsa_*at import time, since identical labels in mlkem-native and mldsa-native (e.g.Lntt_layer123_start) made the FIPS delocator panic.Upstream now namespaces local labels per project (pq-code-package/mlkem-native#1813:
Lmlk_*; pq-code-package/mldsa-native#1310:Lmld_*), so the workaround is dropped.Filenames
Upstream has renamed backend .S files to have globally unique names -- e.g. removing clashes between
ntt_aarch64_asm.Sin both mlkem-native and mldsa-native -- which broke the import into s2n-tls.Compatibility layer
MLD_SYS_CAP_X86_64_AVX2), still mapped to AWS-LC'sCRYPTO_is_{AVX2,NEON}_capable().sig_lenargument from sign/verify (signatures are fixed-length); theml_dsa.cwrappers are adjusted accordingly. The wrappers now also validatemu_lenbefore calling mldsa-native, which expects a fixed 64-bytemubuffer.Build
file(GLOB ... CONFIGURE_DEPENDS), matching mldsa, to absorb upstream.Srenames.GITHUB_SERVER_URLoverridable for imports from local mirrors or forks.Verification
All imported assembly has HOL Light functional-correctness and safety proofs upstream. All assembly except the rejection samplers additionally has proofs of secret-independent timing (the samplers are constant-time as well, but not yet proved so).
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.