1.98 memory usage regression

[sveitser]

Overall memory requirement for compiling our espresso-node crate roughly doubles from 10GB to 20GB from 1.97.1 to 1.98. The issue was discovered because our CI runners have 16GB RAM and fail to build on 1.98

Example: https://github.com/EspressoSystems/espresso-network/actions/runs/32478615863/job/96760043575

Exit code: 143

LLM assisted bisection yielded #156417 as the cause. I can provide more of the LLMs outputs if useful.

Code

I tried this code:

git clone https://github.com/EspressoSystems/espresso-network
cd espresso-network
RUSTC_BOOTSTRAP=espresso_network cargo rustc -p espresso-node --lib -- -Ztime-passes

I expected to see this happen: compiles with roughly the same memory requirement as 1.97.1

$ RUSTC_BOOTSTRAP=espresso_network cargo rustc -p espresso-node --lib -- -Ztime-passes
..
time:   0.001; rss: 1372MB -> 1372MB (   +0MB)	monomorphization_collector_root_collections
time:  57.142; rss: 1372MB -> 3067MB (+1695MB)	monomorphization_collector_graph_walk
time:   3.256; rss: 3061MB -> 3344MB ( +283MB)	partition_and_assert_distinct_symbols
time:  71.973; rss: 1358MB -> 3291MB (+1933MB)	generate_crate_metadata
time:   0.000; rss: 3291MB -> 3291MB (   +0MB)	find_cgu_reuse
time:  16.406; rss: 3291MB -> 3838MB ( +547MB)	codegen_to_LLVM_IR
time:  16.450; rss: 3291MB -> 3838MB ( +547MB)	codegen_crate
time:   0.000; rss: 3839MB -> 3839MB (   +0MB)	incr_comp_persist_dep_graph
time:  14.824; rss: 3666MB -> 3834MB ( +169MB)	LLVM_passes
..

Instead, this happened: Uses significantly more memory on 1.98

$ RUSTC_BOOTSTRAP=espresso_network cargo rustc -p espresso-node --lib -- -Ztime-passes
..
time:   0.001; rss: 1345MB -> 1345MB (   +0MB)	monomorphization_collector_root_collections
time:  65.445; rss: 1345MB -> 14974MB (+13629MB)	monomorphization_collector_graph_walk
time:   3.033; rss: 14987MB -> 15276MB ( +289MB)	partition_and_assert_distinct_symbols
time:  78.328; rss: 1350MB -> 15220MB (+13870MB)	generate_crate_metadata
time:   0.000; rss: 15220MB -> 15220MB (   +0MB)	find_cgu_reuse
time:  16.254; rss: 15220MB -> 17343MB (+2123MB)	codegen_to_LLVM_IR
time:  16.256; rss: 15220MB -> 17344MB (+2124MB)	codegen_crate
time:   0.000; rss: 17344MB -> 17344MB (   +0MB)	serialize_dep_graph
time:  18.253; rss: 15418MB -> 6046MB (-9372MB)	LLVM_passes
..

Version it worked on

rustc 1.97.1 (8bab26f4f 2026-07-14)
binary: rustc
commit-hash: 8bab26f4f68e0e26f0bb7960be334d5b520ea452
commit-date: 2026-07-14
host: x86_64-unknown-linux-gnu
release: 1.97.1
LLVM version: 22.1.6

Version with regression

rustc 1.98.0 (88d9e12ae 2026-08-18)
binary: rustc
commit-hash: 88d9e12ae178fab0fb5cc050a94da85685d449ea
commit-date: 2026-08-18
host: x86_64-unknown-linux-gnu
release: 1.98.0
LLVM version: 22.1.8

Backtrace

no crash

[apiraino]

@sveitser the licensing of your project is a bit unclear:

[...] While we plan to adopt an open source license, we have not yet selected one. As such, all rights are reserved for the time being. Please reach out to us if you have thoughts on licensing.

I assume that this text is just not updated? Asking just to be sure we don't risk any license infringement by downloading the code and compiling to try to reproduce the issue

[sveitser]

@apiraino thanks for pointing out this issue with our lack of official license. We simply never got around to it but it's being worked on now. I just can't really say when the license will be available/decided.

In the meantime I think the below LLM generated snippet could serve as a repro instead. It triggers time and memory regressions at the same releases and nightly commits as the original code. It's inspired by the NodeType trait that is used everywhere in the code base.

$ RUSTC_BOOTSTRAP=1 cargo +1.97.1 rustc -- -Ztime-passes 2>&1 | grep collector_graph_walk
time:   1.557; rss:   65MB ->   68MB (   +3MB)	monomorphization_collector_graph_walk

$ RUSTC_BOOTSTRAP=1 cargo +1.98 rustc -- -Ztime-passes 2>&1 | grep collector_graph_walk
time:   6.785; rss:   68MB ->  369MB ( +301MB)	monomorphization_collector_graph_walk

$ cargo +bisector-ci-b3f7e3204640c7458c9f5096f3e9c5ad9971a51c-x86_64-unknown-linux-gnu rustc -- -Ztime-passes 2>&1 | grep collector_graph_walk
time:   1.869; rss:   66MB ->   70MB (   +4MB)	monomorphization_collector_graph_walk

$ cargo +bisector-ci-688ab44c7439a226d792990f81cd8df0603747d8-x86_64-unknown-linux-gnu rustc -- -Ztime-passes 2>&1 | grep collector_graph_walk
time:   4.076; rss:   66MB ->  368MB ( +303MB)	monomorphization_collector_graph_walk

#![allow(dead_code)]

pub trait Bound {}
pub trait Extra {}
impl<T: Bound> Extra for T {}

pub struct Pair<A, B>(A, B);
impl<A: Bound, B: Bound> Bound for Pair<A, B> {}

pub struct P<const I: usize, const S: usize>;
impl<const I: usize, const S: usize> Bound for P<I, S> {}

pub struct Node<const I: usize>;

pub struct W8<A, B, C, D, E, F, G, H>(A, B, C, D, E, F, G, H);
pub type L1<T> = W8<T, T, T, T, T, T, T, T>;
pub type L2<T> = W8<L1<T>, L1<T>, L1<T>, L1<T>, L1<T>, L1<T>, L1<T>, L1<T>>;
pub type L3<T> = W8<L2<T>, L2<T>, L2<T>, L2<T>, L2<T>, L2<T>, L2<T>, L2<T>>;
pub type L4<T> = W8<L3<T>, L3<T>, L3<T>, L3<T>, L3<T>, L3<T>, L3<T>, L3<T>>;
pub type L5<T> = W8<L4<T>, L4<T>, L4<T>, L4<T>, L4<T>, L4<T>, L4<T>, L4<T>>;

pub trait NodeType: 'static {
    type A0: Bound + Extra + Send + Sync;
    type A1: Bound + Extra + Send + Sync;
    type A2: Bound + Extra + Send + Sync;
    type A3: Bound + Extra + Send + Sync;
    type A4: Bound + Extra + Send + Sync;
}

impl<const I: usize> NodeType for Node<I> {
    type A0 = P<I, 0>;
    type A1 = P<I, 1>;
    type A2 = P<I, 2>;
    type A3 = P<I, 3>;
    type A4 = P<I, 4>;
}

macro_rules! level {
    ($name:ident, $method:ident, [$(($a:ident, $b:ident)),*], $auto:ident $(, $sup:ident)?) => {
        pub trait $name<T: NodeType> $(: $sup<T>)?
        where
            T::A0: Bound + Extra, T::A1: Bound + Extra, T::A2: Bound + Extra,
            T::A3: Bound + Extra, T::A4: Bound + Extra,
            $( Pair<T::$a, T::$b>: Bound + Extra, )*
            L5<T::$auto>: Send + Sync,
        {
            fn $method(&self) -> usize { 0 }
        }

        impl<X, T: NodeType> $name<T> for X
        where
            X: 'static $(+ $sup<T>)?,
            T::A0: Bound + Extra, T::A1: Bound + Extra, T::A2: Bound + Extra,
            T::A3: Bound + Extra, T::A4: Bound + Extra,
            $( Pair<T::$a, T::$b>: Bound + Extra, )*
            L5<T::$auto>: Send + Sync,
        {}
    };
}

// Each level states different bounds so the goals do not dedupe across the chain.
level!(S0, s0, [(A0, A1), (A2, A3), (A4, A0)], A0);
level!(S1, s1, [(A0, A2), (A1, A3), (A4, A1)], A1, S0);
level!(Api, m0, [(A0, A3), (A1, A4), (A2, A0)], A2, S1);

macro_rules! build_vtables {
    ($($i:literal)*) => {
        $( { let _d: &dyn Api<Node<$i>> = &0u8; } )*
    };
}

fn main() {
    build_vtables!(0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19);
}

[apiraino]

cc @jakubadamw do you anything relevant to #156417 about this regression?

thanks

[shadaj]

We are seeing a similar issue, with builds succeeding on 1.96 but failing on latest nightly: https://github.com/hydro-project/hydro/actions/runs/32769619291/job/97963449196?pr=3152

[theemathas]

cc @lcnr

[jakubadamw]

cc @jakubadamw do you anything relevant to #156417 about this regression?

@apiraino, we did see some negative results for max-rss in that PR, but my reading of it was that it’s rather a noisy metric, so I didn’t take much notice of it. To be fair, the PR addressed an ICE (in what was deemed to be logically the preferred way), but now I can see another issue has been filed indicating one of the ICEs has merely moved from typechecking to codegen. So unless anyone else has an immediate idea on how to alleviate the memory usage peak (I sadly don’t off the top of my head), we could consider reverting this (again), seeing things are a little patchy over all here.

[sveitser]

In case it's useful llm assisted further shrinking of the repro lead to the much smaller lib.rs below

#![allow(dead_code)]

pub struct P;
pub struct Node;

pub type L1<T> = (T, T, T, T);
pub type L2<T> = (L1<T>, L1<T>, L1<T>, L1<T>);
pub type L3<T> = (L2<T>, L2<T>, L2<T>, L2<T>);
pub type L4<T> = (L3<T>, L3<T>, L3<T>, L3<T>);
pub type L5<T> = (L4<T>, L4<T>, L4<T>, L4<T>);
pub type L6<T> = (L5<T>, L5<T>, L5<T>, L5<T>);
pub type L7<T> = (L6<T>, L6<T>, L6<T>, L6<T>);
pub type L8<T> = (L7<T>, L7<T>, L7<T>, L7<T>);
pub type L9<T> = (L8<T>, L8<T>, L8<T>, L8<T>);

pub trait NodeType {
    type A;
}

impl NodeType for Node {
    type A = P;
}

pub trait Api<T: NodeType>
where
    L9<T::A>: Send,
{
    fn m0(&self) -> usize { 0 }
}

impl<X, T: NodeType> Api<T> for X where L9<T::A>: Send {}

pub fn build_vtable() {
    let _d: &dyn Api<Node> = &0u8;
}

$ cargo +1.97.1 rustc -- -Ztime-passes
time:   0.052; rss:  102MB ->  102MB (   +0MB)	monomorphization_collector_graph_walk

$ cargo +1.98 rustc -- -Ztime-passes
time:   0.361; rss:  106MB ->  148MB (  +42MB)	monomorphization_collector_graph_walk

$ cargo +bisector-ci-b3f7e3204640c7458c9f5096f3e9c5ad9971a51c-x86_64-unknown-linux-gnu rustc -- -Ztime-passes
time:   0.054; rss:  102MB ->  102MB (   +0MB)	monomorphization_collector_graph_walk

$ cargo +bisector-ci-688ab44c7439a226d792990f81cd8df0603747d8-x86_64-unknown-linux-gnu rustc -- -Ztime-passes
time:   0.187; rss:  102MB ->  153MB (  +51MB)	monomorphization_collector_graph_walk

[lcnr]

we now prove L9<P> Send when to codegen <dyn Api<Node>>::m0. I don't really get why this causes a huge max rss issue. Is most of the memory cost NestedGoals in the search graph?

I would prefer to not revert this change as it is the correct behavior and shouldn't be that expensive, the only thing that's expensive is that we prove more stuff with the new trait solver, and that's a regression we should fix in the new trait solver. Looking into this a bit. Yeah with the new solver type_check_crate also takes 385MB for the bigger example and then that just triggers the OOM killer :>

so this seems like a limitation of the new trait solver, mind testing your crate with the latest nightly and check whether it also significantly regresses there? #160895

[sveitser]

@lcnr For the last small repro (in #161660 (comment)), with latest nightly it runs out of memory (at ~33GB, about 30 seconds) during the graph walk I believe.

cargo +nightly --version --verbose
cargo 1.100.0-nightly (e8cb624d5 2026-08-22)
release: 1.100.0-nightly
commit-hash: e8cb624d5701824f46a2ec5873cfd59ee3d2f66c
commit-date: 2026-08-22
host: x86_64-unknown-linux-gnu
libgit2: 1.9.6 (sys:0.21.0 vendored)
libcurl: 8.21.0-DEV (sys:0.4.90+curl-8.21.0 vendored ssl:OpenSSL/3.6.3)
ssl: OpenSSL 3.6.3 9 Jun 2026
os: NixOS 26.11.0 (zokor) [64-bit]

/usr/bin/env time cargo +nightly rustc -- -Ztime-passes
   Compiling vtable-repro v0.1.0 (/home/lulu/r/sveitser/ntrepro-min)
time:   0.000; rss:   29MB ->   30MB (   +1MB)	parse_crate
time:   0.009; rss:   31MB ->   31MB (   +0MB)	incr_comp_prepare_session_directory
time:   0.000; rss:   31MB ->   31MB (   +0MB)	incr_comp_garbage_collect_session_directories
time:   0.000; rss:   35MB ->   35MB (   +0MB)	crate_injection
time:   0.002; rss:   35MB ->   41MB (   +6MB)	expand_crate
time:   0.000; rss:   41MB ->   41MB (   +0MB)	check_unused_macros
time:   0.002; rss:   35MB ->   41MB (   +6MB)	macro_expand_crate
time:   0.000; rss:   41MB ->   42MB (   +0MB)	AST_validation
time:   0.000; rss:   42MB ->   42MB (   +0MB)	finalize_imports
time:   0.000; rss:   42MB ->   42MB (   +0MB)	compute_effective_visibilities
time:   0.000; rss:   42MB ->   42MB (   +0MB)	finalize_macro_resolutions
time:   0.000; rss:   42MB ->   44MB (   +2MB)	late_resolve_crate
time:   0.000; rss:   44MB ->   44MB (   +0MB)	resolve_check_unused
time:   0.000; rss:   44MB ->   44MB (   +0MB)	resolve_postprocess
time:   0.000; rss:   42MB ->   44MB (   +2MB)	resolve_crate
time:   0.000; rss:   44MB ->   44MB (   +0MB)	write_dep_info
time:   0.000; rss:   44MB ->   44MB (   +0MB)	complete_gated_feature_checking
time:   0.000; rss:   45MB ->   46MB (   +0MB)	looking_for_entry_point
time:   0.000; rss:   46MB ->   46MB (   +0MB)	check_externally_implementable_items
time:   0.000; rss:   46MB ->   46MB (   +0MB)	unused_lib_feature_checking
time:   0.000; rss:   45MB ->   46MB (   +1MB)	misc_checking_1
time:   1.358; rss:   46MB ->  169MB ( +122MB)	coherence_checking
time:   1.671; rss:   46MB ->  186MB ( +140MB)	type_check_crate
time:   0.001; rss:  186MB ->  191MB (   +5MB)	MIR_borrow_checking
time:   0.001; rss:  191MB ->  192MB (   +1MB)	module_lints
time:   0.001; rss:  191MB ->  192MB (   +1MB)	lint_checking
time:   0.005; rss:  192MB ->  192MB (   +0MB)	privacy_checking_modules
time:   0.000; rss:  192MB ->  192MB (   +0MB)	check_lint_expectations
time:   0.014; rss:  191MB ->  192MB (   +1MB)	misc_checking_3
time:   0.000; rss:  192MB ->  192MB (   +0MB)	generate_crate_metadata_stub
time:   0.000; rss:  191MB ->  191MB (   +0MB)	monomorphization_collector_root_collections
error: could not compile `vtable-repro` (lib)

Caused by:
  process didn't exit successfully: `/home/lulu/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/rustc --crate-name vtable_repro --edition=2021 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=233 --crate-type lib --emit=dep-info,metadata,link -Z embed-metadata=no -C embed-bitcode=no -C debuginfo=2 --check-cfg 'cfg(docsrs,test)' --check-cfg 'cfg(feature, values())' -C metadata=685e0f72a2fc31c3 -C extra-filename=-69cfbe61a693bd1b --out-dir /home/lulu/r/sveitser/ntrepro-min/target/debug/build/vtable-repro/69cfbe61a693bd1b/out -C incremental=/home/lulu/r/sveitser/ntrepro-min/target/debug/incremental -Ztime-passes` (signal: 15, SIGTERM: termination signal)
Command exited with non-zero status 101

20.38user 9.87system 0:30.46elapsed 99%CPU (0avgtext+0avgdata 33959412maxresident)k
384inputs+9outputs (2major+8618343minor)pagefaults 0swaps

Aug 27 11:17:21 bix earlyoom[1693]: sending SIGTERM to process 146052 uid 1000 "rustc": oom_score 1444, oom_score_adj 200, VmRSS 33088 MiB, cmdline "/home/lulu/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/rustc --crate-na>

For the original crate I opened the issue the same step runs out of memory.

..
time:   0.048; rss: 4031MB -> 4031MB (   +0MB)	lint_checking
time:   0.036; rss: 4031MB -> 4031MB (   +0MB)	privacy_checking_modules
time:   0.103; rss: 4031MB -> 4031MB (   +0MB)	misc_checking_3
time:   0.001; rss: 4031MB -> 4031MB (   +0MB)	generate_crate_metadata_stub
time:   0.001; rss: 4662MB -> 4662MB (   +0MB)	monomorphization_collector_root_collections
warning: `espresso-node` (lib) generated 1 warning
error: could not compile `espresso-node` (lib); 1 warning emitted

[lcnr]

nice, thank you! already have an ongoing convo about likely the same issue in bevy_render #161748 https://rust-lang.zulipchat.com/#narrow/channel/364551-t-types.2Ftrait-system-refactor/topic/bevy_render-0.2E19.2E1/with/619470267