Skip to content

Re-import mlkem-native and mldsa-native; drop custom meta headers - #3367

Open
hanno-becker wants to merge 1 commit into
aws:mainfrom
hanno-becker:reimport-mlkem-mldsa-drop-shadow-meta
Open

Re-import mlkem-native and mldsa-native; drop custom meta headers#3367
hanno-becker wants to merge 1 commit into
aws:mainfrom
hanno-becker:reimport-mlkem-mldsa-drop-shadow-meta

Conversation

@hanno-becker

@hanno-becker hanno-becker commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

This re-imports mlkem-native and mldsa-native from their respective main branches (mlkem-native 61c831345, mldsa-native 9a180a96b).

Meta headers

This PR removes the three AWS-LC-owned "shadow" backend meta headers: ml_kem/mlkem_aarch64_meta.h, ml_dsa/mldsa_aarch64_meta.h, and ml_dsa/mldsa_x86_64_meta.h. The two AArch64 shadows (added in #3353)
duplicated the vendored meta.h to add a runtime NEON gate with C fallback (reachable via OPENSSL_armcap=0); upstream now provides this gate itself (MLK_SYS_CAP_NEON / MLD_SYS_CAP_AARCH64_NEON), making the
shadows exact duplicates. All backends consume the vendored meta.h as-is.

Full x86_64 ML-DSA backend

The x86_64 mldsa shadow trimmed the vendored meta.h to the assembly-backed subset that AWS-LC imported. The remaining AVX2 C-intrinsic operations have since been rewritten and proved in assembly upstream, so we
now import the full x86_64 backend.

Assembly local labels

Earlier revisions of this PR rewrote mldsa's local assembly labels to Lmldsa_* at import time, since identical labels in mlkem-native and mldsa-native (e.g. Lntt_layer123_start) made the FIPS delocator panic.
Upstream now namespaces local labels per project (pq-code-package/mlkem-native#1813: Lmlk_*; pq-code-package/mldsa-native#1310: Lmld_*), so the workaround is dropped.

Filenames

Upstream has renamed backend .S files to have globally unique names -- e.g. removing clashes between ntt_aarch64_asm.S in both mlkem-native and mldsa-native -- which broke the import into s2n-tls.

Compatibility layer

  • Capability func: the custom configs revert to the vendored enum type (mldsa's AVX2 enumerator is now MLD_SYS_CAP_X86_64_AVX2), still mapped to AWS-LC's CRYPTO_is_{AVX2,NEON}_capable().
  • ML-DSA API: upstream dropped the sig_len argument from sign/verify (signatures are fixed-length); the ml_dsa.c wrappers are adjusted accordingly. The wrappers now also validate mu_len before calling mldsa-native, which expects a fixed 64-byte mu buffer.
  • The mldsa-native update also brings upstream hardening: keygen and signing outputs are zeroized on failure.

Build

  • The mlkem assembly lists in crypto/fipsmodule/CMakeLists.txt switch to file(GLOB ... CONFIGURE_DEPENDS), matching mldsa, to absorb upstream .S renames.
  • Both importer.sh scripts make GITHUB_SERVER_URL overridable for imports from local mirrors or forks.

Verification

All imported assembly has HOL Light functional-correctness and safety proofs upstream. All assembly except the rejection samplers additionally has proofs of secret-independent timing (the samplers are constant-time as well, but not yet proved so).


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.

@hanno-becker
hanno-becker force-pushed the reimport-mlkem-mldsa-drop-shadow-meta branch from 4a4b530 to 16c4085 Compare July 23, 2026 05:04
@codecov-commenter

codecov-commenter commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.00000% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.47%. Comparing base (6e339a1) to head (bd45929).

Files with missing lines Patch % Lines
crypto/fipsmodule/ml_dsa/mldsa/sign.c 51.78% 27 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3367      +/-   ##
==========================================
+ Coverage   78.41%   78.47%   +0.05%     
==========================================
  Files         695      696       +1     
  Lines      124223   124348     +125     
  Branches    17267    17303      +36     
==========================================
+ Hits        97413    97584     +171     
+ Misses      25886    25840      -46     
  Partials      924      924              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jakemas

jakemas commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Reviewed, this covers the same ML-DSA x86 re-import and sig_len handling as my #3357 (which it supersedes, and with a cleaner verify guard), and since it also brings mlkem-native along I'm happy to go with this PR instead.

mkannwischer added a commit to pq-code-package/mldsa-native that referenced this pull request Jul 25, 2026
Update the pinned AWS-LC integration test from v5.1.0 to v5.4.0.

The importer/glue in released AWS-LC predates the current mldsa-native
API (dropped sig_len, renamed capability enums to MLD_SYS_CAP_X86_64_AVX2
and MLD_SYS_CAP_AARCH64_NEON). aws/aws-lc#3367 is the upstream re-import
that makes AWS-LC compatible again, but it is not yet in any release, so
its integration-layer changes are applied here as a pre-import patch:
update importer.sh, ml_dsa.c, mldsa_native_config.h and
mldsa_native_backend.h, and drop the custom mldsa_x86_64_meta.h /
mldsa_aarch64_meta.h in favour of the vendored meta.h imported verbatim.

This fixes our CI integration tests against AWS-LC main.

Also exclude patch files from scripts/autogen's macro-typo check, since
they carry old macro names on their context and deletion lines.

- Fixes #1300

Signed-off-by: Matthias J. Kannwischer <matthias@zerorisc.com>
mkannwischer added a commit to pq-code-package/mldsa-native that referenced this pull request Jul 26, 2026
Update the pinned AWS-LC integration test from v5.1.0 to v5.4.0.

The importer/glue in released AWS-LC predates the current mldsa-native
API (dropped sig_len, renamed capability enums to MLD_SYS_CAP_X86_64_AVX2
and MLD_SYS_CAP_AARCH64_NEON). aws/aws-lc#3367 is the upstream re-import
that makes AWS-LC compatible again, but it is not yet in any release, so
its integration-layer changes are applied here as a pre-import patch:
update importer.sh, ml_dsa.c, mldsa_native_config.h and
mldsa_native_backend.h, and drop the custom mldsa_x86_64_meta.h /
mldsa_aarch64_meta.h in favour of the vendored meta.h imported verbatim.

This fixes our CI integration tests against AWS-LC main.

Also exclude patch files from scripts/autogen's macro-typo check, since
they carry old macro names on their context and deletion lines.

- Fixes #1300

Signed-off-by: Matthias J. Kannwischer <matthias@zerorisc.com>
@hanno-becker
hanno-becker force-pushed the reimport-mlkem-mldsa-drop-shadow-meta branch from 16c4085 to 8ce405c Compare July 27, 2026 04:48
@hanno-becker

Copy link
Copy Markdown
Contributor Author

CI legitimately failing because the code tries to use the PPC64LE backend which is not imported.

@hanno-becker

Copy link
Copy Markdown
Contributor Author

mldsa-native v2 is imminent, so we should hold back merge until this PR is updated.

github-actions[bot]

This comment was marked as spam.

@jakemas

jakemas commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

mldsa-native v2 is imminent, so we should hold back merge until this PR is updated.

Looks like v2 tasks are now complete, shall we do a fresh import here. I am happy otherwise.

jakemas
jakemas previously approved these changes Aug 5, 2026
torben-hansen
torben-hansen previously approved these changes Aug 6, 2026
@rod-chapman

Copy link
Copy Markdown

I think this PR needs to be updated to pull the v2.0.0 tags/commits of mldsa-native and mlkem-native?

This commit re-imports mlkem-native (v2.0.0, d1b2fe782) and
mldsa-native (v2.0.0, 834a90d5e), replacing mlkem-native v1.1.0
(d2cae2be) and mldsa-native 08d40f9403 as previously vendored.

With the re-import, we remove the three custom backend meta headers
previously owned and used by AWS-LC (ml_kem/mlkem_aarch64_meta.h,
ml_dsa/mldsa_aarch64_meta.h, and ml_dsa/mldsa_x86_64_meta.h):
- The AArch64 shadows existed to add a runtime NEON gate with a C
  fallback. Now, upstream gates NEON in the vendored meta.h itself.
- The x86_64 mldsa shadow trimmed the backend to the assembly-backed
  subset. Upstream has since converted the remaining AVX2 C-intrinsic
  operations to proven assembly, so the full backend is imported and
  its meta.h is used as-is.

The commit also removes a previous label-renaming at import time that
was in place to avoid a name clash between mlkem and mldsa labels.
Upstream has since made the labels globally unique (mlkem-native
PR#1813 and mldsa-native PR#1310).

The commit also removes a previous file-renaming at import time that
was in place to avoid a name clash between the mlkem and mldsa assembly
files, which are compiled into the same FIPS module. Upstream has since
prefixed the backend assembly filenames per project
(mlkem_*_asm.S and mldsa_*_asm.S), so the basenames are globally unique
and the individual .S files can again be built directly.

This commit also updates custom configurations to use the vendored
capability enum while retaining AWS-LC mappings to
CRYPTO_is_AVX2_capable() and CRYPTO_is_NEON_capable(). Both projects
now qualify the enumerators by architecture, so mlkem_native_config.h
uses MLK_SYS_CAP_X86_64_AVX2 and MLK_SYS_CAP_AARCH64_NEON, and
mldsa_native_config.h the MLD_ equivalents. Without this, the build
breaks outright, as the unqualified MLK_SYS_CAP_AVX2 / MLK_SYS_CAP_NEON
no longer exist.

The ML-KEM assembly source lists in crypto/fipsmodule/CMakeLists.txt
now use file(GLOB ... CONFIGURE_DEPENDS), matching ML-DSA. This
automatically absorbs upstream assembly-file additions, removals, and
renames without corresponding CMake changes. Both importer scripts also
allow GITHUB_SERVER_URL to be overridden for imports from local mirrors
or forks.

mlkem-native v2.0.0 is a breaking release. Beyond the capability enum
above, it removes the deprecated MLK_CONFIG_API_* configuration
mechanism and the SUPERCOP API (crypto_kem_*, CRYPTO_SECRETKEYBYTES,
CRYPTO_BYTES, ...), both of which the vendored v1.1.0 still provided.
AWS-LC configures mlkem-native through the recommended config file and
never consumed the SUPERCOP names, so neither removal requires a change
here; the latter does mean the FIPS module is no longer exposed to those
generically named macros.

Both projects also refine their error codes. mlkem-native replaces the
single generic MLK_ERR_FAIL with MLK_ERR_INVALID_PK, MLK_ERR_INVALID_SK
and MLK_ERR_PCT_FAIL for the FIPS 203 modulus check, hash check, and
Pairwise Consistency Test respectively. mldsa-native adds
MLD_ERR_SIGNING_PAUSED, MLD_ERR_INVALID_SIGNATURE, MLD_ERR_INVALID_KEY,
MLD_ERR_PCT_FAIL and MLD_ERR_INVALID_ARG to the four codes previously
vendored.

The mldsa refinement is inconsequential here, because ml_dsa.c already
normalised every mldsa-native return code to 1-on-success and never
propagated the code itself. ml_kem.c had no such insulation: it returned
mlkem-native's code verbatim, so the mlkem renumbering would silently
have changed the values observable through ml_kem.h -- e.g.
encapsulation against a malformed public key returning -4 rather than
-1. See the return-value conventions below.

Two further mldsa-native changes do not affect this integration:
- MLD_CONFIG_NO_RANDOMIZED_API now gates keypair(), signature() and
  signature_extmu(), and MLD_CONFIG_CORE_API_ONLY is newly declared
  incompatible with MLD_CONFIG_KEYGEN_PCT. AWS-LC sets neither option.
- Following the FIPS 204 errata @[FIPS204_UPDATES], the FIPS-compliant
  lower bound on MLD_CONFIG_MAX_SIGNING_ATTEMPTS rises from 814 to 821.
  AWS-LC does not set the option, so the fallback bound
  MLD_MAX_KAPPA / MLDSA_L applies -- 16382, 13106 and 9361 attempts for
  ML-DSA-44/65/87 -- comfortably above the new bound.

API changes: Upstream mldsa-native has changed its sign and verify API
to no longer accept a sig_len parameter. Instead, it assumes that the
signature length is the static length associated with the chosen variant
of ML-DSA. When moving from a higher-level API with variable-length
signatures to mldsa-native's lower-level API, those length checks need
to happen prior to calling into mldsa-native. The commit adds this in
ml_dsa.c accordingly.

The commit also fixes a minor API issue in the wrapper ml_dsa.c around
mldsa-native whereby the nominally variable mu_len parameter was passed
through to mldsa-native unchecked even though mldsa-native expects a buffer
of static size 64 bytes. This was not exploitable since the next higher
layer does check the length, rejecting message_len != pqdsa->digest_len in
pkey_pqdsa_sign_generic() and pkey_pqdsa_verify_generic()
(p_pqdsa.c:191 and :282). Still so long as ml_dsa.h exposes an API with
variable-length mu, it should check for the expected length.

The mldsa-native update also ensures that key generation and signing
outputs are zeroized on failure.

Return-value conventions: Neither ml_kem.h nor ml_dsa.h documented what
their functions return, and the two use opposite conventions -- ml_dsa.h
returns 1 on success per AWS-LC's usual convention (API-CONVENTIONS.md),
while ml_kem.h inherited 0-on-success from mlkem-native. Both headers now
state their convention and define named constants (ML_KEM_SUCCESS /
ML_KEM_FAILURE, ML_DSA_SUCCESS / ML_DSA_FAILURE), each warning about the
other.

Each wrapper gains a static inline conversion function, ml_kem_result()
and ml_dsa_result(), and every site that translates a vendored return
code now goes through it. In ml_dsa.c this is behaviour-preserving, as
the open-coded conversions it replaces already normalised to
1-on-success. In ml_kem.c it stops the pass-through described above:
every failure now yields ML_KEM_FAILURE, so upstream's numbering is no
longer part of AWS-LC's surface and future refinements are absorbed
without further change. This does alter ML-KEM's failure values (the
argument-rejection path in check_pk/check_sk previously returned -1,
colliding with the value upstream now reserves for MLK_ERR_FAIL), but no
caller is affected: crypto/fipsmodule/kem/kem.c, self_check.c,
modulewrapper.cc and ml_kem_test.cc all test only for zero versus
non-zero.

The two files that call both APIs, self_check.c and modulewrapper.cc,
must therefore use inverted success tests for the two modules. Both now
carry a note to that effect at the relevant call sites, since an
inverted test in a self-test or KAT would make the check pass
unconditionally rather than fail.

The PCT comments in ml_kem.c and ml_dsa.c now record why PCT failure is
the only *reachable* keygen failure, so that AWS_LC_FIPS_failure() is
not reached for an unrelated condition.

All imported assembly has HOL Light functional-correctness and safety
proofs upstream. All assembly except the rejection samplers additionally
has proofs of secret-independent timing.

Signed-off-by: Hanno Becker <beckphan@amazon.co.uk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants