Cross-compilation default settings do not work, only show obscure error messages

[RalfJung]

I am trying to use the GHA aarch64-unknown-linux-gnu runners to build and run my code for armv7-unknown-linux-gnueabihf. Since my code depends on the rustc crates, I cannot do actual cross-compilation. Instead, I am installing the armv7-unknown-linux-gnueabihf version of rustc on that system. rustc itself runs successfully since I installed enough multiarch packages. However, Rust fails to actually build any binaries in this setup (specifically, it fails to build the build scripts):

  error: linking with `cc` failed: exit status: 1
    |
    = note:  "cc" "/tmp/rustcWH48QS/symbols.o" "<4 object files omitted>" "-Wl,--as-needed" "-Wl,-Bstatic" "<sysroot>/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/{libstd-*,libpanic_unwind-*,libobject-*,libmemchr-*,libaddr2line-*,libgimli-*,librustc_demangle-*,libstd_detect-*,libhashbrown-*,librustc_std_workspace_alloc-*,libminiz_oxide-*,libadler2-*,libunwind-*,libcfg_if-*,liblibc-*,librustc_std_workspace_core-*,liballoc-*,libcore-*,libcompiler_builtins-*}.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-L" "/tmp/rustcWH48QS/raw-dylibs" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "<sysroot>/lib/rustlib/armv7-unknown-linux-gnueabihf/lib" "-o" "/home/runner/work/miri/miri/target/release/build/libc-3375c08ac196722c/build_script_build-3375c08ac196722c" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,--strip-debug" "-nodefaultlibs" "-Wl,-rpath,<sysroot>/lib/rustlib/armv7-unknown-linux-gnueabihf/lib"
    = note: some arguments are omitted. use `--verbose` to show all linker arguments
    = note: /usr/bin/ld: /tmp/rustcWH48QS/symbols.o: error adding symbols: file in wrong format
            collect2: error: ld returned 1 exit status
            
  
  error: could not compile `libc` (build script) due to 1 previous error
  warning: build failed, waiting for other jobs to finish...
  error: linking with `cc` failed: exit status: 1
    |
    = note:  "cc" "/tmp/rustcwYoYoY/symbols.o" "<4 object files omitted>" "-Wl,--as-needed" "-Wl,-Bstatic" "<sysroot>/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/{libstd-*,libpanic_unwind-*,libobject-*,libmemchr-*,libaddr2line-*,libgimli-*,librustc_demangle-*,libstd_detect-*,libhashbrown-*,librustc_std_workspace_alloc-*,libminiz_oxide-*,libadler2-*,libunwind-*,libcfg_if-*,liblibc-*,librustc_std_workspace_core-*,liballoc-*,libcore-*,libcompiler_builtins-*}.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-L" "/tmp/rustcwYoYoY/raw-dylibs" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "<sysroot>/lib/rustlib/armv7-unknown-linux-gnueabihf/lib" "-o" "/home/runner/work/miri/miri/target/release/build/zerocopy-f22a37c7a8ae1a41/build_script_build-f22a37c7a8ae1a41" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,--strip-debug" "-nodefaultlibs" "-Wl,-rpath,<sysroot>/lib/rustlib/armv7-unknown-linux-gnueabihf/lib"
    = note: some arguments are omitted. use `--verbose` to show all linker arguments
    = note: /usr/bin/ld: /tmp/rustcwYoYoY/symbols.o: error adding symbols: file in wrong format
            collect2: error: ld returned 1 exit status
            
  
  error: could not compile `zerocopy` (build script) due to 1 previous error

I have no idea about any of the internals here, but I suspect that cc needs to be told that this is a cross-compilation situation, since the host cc will by default build for aarch64-unknown-linux-gnu but rustc is building for armv7-unknown-linux-gnueabihf. I tried to set the CC env var to arm-linux-gnueabihf-gcc but that changed nothing.

[RalfJung]

@bjorn3 do you happen to have any idea about this one?

[RalfJung]

If I compare this with a similar situation on x86, where I can also have a 32bit rustc installed on a 64bit system, the cc invocation there begins with "cc" "-m32" .... So there seems to be some special-casing going on here? But AFAIK -m32 is part of "multilib" which is not the same as "multiarch"; I don't know much about the difference though.

[bjorn3]

Did you try setting the linker using -Clinker or target."<target>".linker in .cargo/config.toml to arm-linux-gnueabihf-gcc?

[RalfJung]

No, I didn't know about that flag.

Hm, now I have to figure out how to plumb that though CI.^^

[RalfJung]

No, that does not seem to help:

   error: linking with `arm-linux-gnueabihf-gcc` failed: exit status: 1
    |
    = note:  "arm-linux-gnueabihf-gcc" "/tmp/rustcTO8wi0/symbols.o" "<3 object files omitted>" "-Wl,--as-needed" "-Wl,-Bstatic" "<sysroot>/lib/rustlib/aarch64-unknown-linux-gnu/lib/{libstd-*,libpanic_unwind-*,libobject-*,libmemchr-*,libaddr2line-*,libgimli-*,librustc_demangle-*,libstd_detect-*,libhashbrown-*,librustc_std_workspace_alloc-*,libminiz_oxide-*,libadler2-*,libunwind-*,libcfg_if-*,liblibc-*,librustc_std_workspace_core-*,liballoc-*,libcore-*,libcompiler_builtins-*}.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-L" "/tmp/rustcTO8wi0/raw-dylibs" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "<sysroot>/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-o" "/home/runner/work/miri/miri/./miri-script/target/debug/build/proc-macro2-7182a73e43762f05/build_script_build-7182a73e43762f05" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs"
    = note: some arguments are omitted. use `--verbose` to show all linker arguments
    = note: /tmp/rustcTO8wi0/symbols.o: file not recognized: file format not recognized
            collect2: error: ld returned 1 exit status
            
  error: could not compile `proc-macro2` (build script) due to 1 previous error
  error: linking with `arm-linux-gnueabihf-gcc` failed: exit status: 1
    |
    = note:  "arm-linux-gnueabihf-gcc" "/tmp/rustcEGt9US/symbols.o" "<5 object files omitted>" "-Wl,--as-needed" "-Wl,-Bstatic" "<sysroot>/lib/rustlib/aarch64-unknown-linux-gnu/lib/{libstd-*,libpanic_unwind-*,libobject-*,libmemchr-*,libaddr2line-*,libgimli-*,librustc_demangle-*,libstd_detect-*,libhashbrown-*,librustc_std_workspace_alloc-*,libminiz_oxide-*,libadler2-*,libunwind-*,libcfg_if-*,liblibc-*,librustc_std_workspace_core-*,liballoc-*,libcore-*,libcompiler_builtins-*}.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-L" "/tmp/rustcEGt9US/raw-dylibs" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "<sysroot>/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-o" "/home/runner/work/miri/miri/./miri-script/target/debug/build/libc-dc9006c4b3194774/build_script_build-dc9006c4b3194774" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs"
    = note: some arguments are omitted. use `--verbose` to show all linker arguments
    = note: /tmp/rustcEGt9US/symbols.o: file not recognized: file format not recognized
            collect2: error: ld returned 1 exit status
            
  error: could not compile `libc` (build script) due to 1 previous error

[bjorn3]

It seems like you are applying it in the arm64 case too.

<sysroot>/lib/rustlib/aarch64-unknown-linux-gnu/lib/[...]

[RalfJung]

Hm yeah I must have copied the wrong output from somewhere... using a config.toml snippet instead of RUSTFLAGS, this does indeed successfully build Miri! :)

Is it expected that I have to set both CC (otherwise, build scripts will fail) and the target-specific linker? I was hoping maybe things could work a bit more out-of-the-box.

[RalfJung]

Here's a way to reproduce this without having a mismatch between rustc and the host system. I did this on my x86-64 Linux system with a standard x86-64 Linux installation of Rust:

• sudo apt install gcc-aarch64-linux-gnu
• rustup target add aarch64-unknown-linux-gnu
• cargo build --target aarch64-unknown-linux-gnu

This yields:

error: linking with `cc` failed: exit status: 1
  |
  = note:  "cc" "/tmp/rustc69iNAd/symbols.o" "<131 object files omitted>" "-Wl,--as-needed" "-Wl,-Bstatic" "/home/r/src/rust/miri/cargo-miri/target/aarch64-unknown-linux-gnu/debug/deps/{libdirectories-abdd0ae0cdcf895c.rlib,libdirs_sys-401f70ca0026bba1.rlib,liboption_ext-022e2db2e1f31d9d.rlib,libcargo_metadata-d6fb093a368f3413.rlib,libserde_json-73afc7df06bc5361.rlib,libitoa-e84befedff8e7106.rlib,libryu-7bfbd8d375c79ae4.rlib,libthiserror-23e61b6cbe987e05.rlib,libcargo_platform-2193fbdc428dc97e.rlib,libcamino-d3bc83047dcc9af5.rlib,librustc_build_sysroot-82a35ccab2d59951.rlib,libwalkdir-a48f0eed355078b6.rlib,libsame_file-b4d2fbe8165f345f.rlib,libtempfile-91d7dc25fb5acfdc.rlib,libgetrandom-97650eaa2fa01a61.rlib,liblibc-9f87ba1e45132860.rlib,libcfg_if-e9107ac3a47999c7.rlib,libfastrand-bb7b9f1376306918.rlib,libonce_cell-3a65c1a4304dd28a.rlib,librustix-56365eb30d292711.rlib,libbitflags-e21a16acd86b2258.rlib,liblinux_raw_sys-0aba8c061cb19518.rlib,libregex-fe2e4b164b50033c.rlib,libregex_automata-7c8938d9c9beddcb.rlib,libaho_corasick-feb17f335c24a437.rlib,libmemchr-e819f6888ee7bd26.rlib,libregex_syntax-e082a37cbee2ba06.rlib,libanyhow-1b745d93dc4d426b.rlib,librustc_version-f692f13aaa5e9e65.rlib,libsemver-f2d7535de6d23db5.rlib,libserde-1f92605b7116c9f0.rlib}.rlib" "<sysroot>/lib/rustlib/aarch64-unknown-linux-gnu/lib/{libstd-*,libpanic_unwind-*,libobject-*,libmemchr-*,libaddr2line-*,libgimli-*,librustc_demangle-*,libstd_detect-*,libhashbrown-*,librustc_std_workspace_alloc-*,libminiz_oxide-*,libadler2-*,libunwind-*,libcfg_if-*,liblibc-*,librustc_std_workspace_core-*,liballoc-*,libcore-*,libcompiler_builtins-*}.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-L" "/tmp/rustc69iNAd/raw-dylibs" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "<sysroot>/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-o" "/home/r/src/rust/miri/cargo-miri/target/aarch64-unknown-linux-gnu/debug/deps/cargo_miri-f5dda67ea17a7da6" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs"
  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: /usr/bin/ld: /home/r/src/rust/miri/cargo-miri/target/aarch64-unknown-linux-gnu/debug/deps/cargo_miri-f5dda67ea17a7da6.00by6ir8r449ic7dn8ydbi3ox.05ubbpu.rcgu.o: Relocations in generic ELF (EM: 183)
          /usr/bin/ld: /home/r/src/rust/miri/cargo-miri/target/aarch64-unknown-linux-gnu/debug/deps/cargo_miri-f5dda67ea17a7da6.00by6ir8r449ic7dn8ydbi3ox.05ubbpu.rcgu.o: Relocations in generic ELF (EM: 183)
          /usr/bin/ld: /home/r/src/rust/miri/cargo-miri/target/aarch64-unknown-linux-gnu/debug/deps/cargo_miri-f5dda67ea17a7da6.00by6ir8r449ic7dn8ydbi3ox.05ubbpu.rcgu.o: Relocations in generic ELF (EM: 183)
          /usr/bin/ld: /home/r/src/rust/miri/cargo-miri/target/aarch64-unknown-linux-gnu/debug/deps/cargo_miri-f5dda67ea17a7da6.00by6ir8r449ic7dn8ydbi3ox.05ubbpu.rcgu.o: Relocations in generic ELF (EM: 183)
          /usr/bin/ld: /home/r/src/rust/miri/cargo-miri/target/aarch64-unknown-linux-gnu/debug/deps/cargo_miri-f5dda67ea17a7da6.00by6ir8r449ic7dn8ydbi3ox.05ubbpu.rcgu.o: Relocations in generic ELF (EM: 183)
          /usr/bin/ld: /home/r/src/rust/miri/cargo-miri/target/aarch64-unknown-linux-gnu/debug/deps/cargo_miri-f5dda67ea17a7da6.00by6ir8r449ic7dn8ydbi3ox.05ubbpu.rcgu.o: error adding symbols: file in wrong format
          collect2: error: ld returned 1 exit status

Manually adding this to the config fixes that:

[target.aarch64-unknown-linux-gnu]
linker = "aarch64-linux-gnu-gcc"

But in an ideal world, I wouldn't have to do this manually...

... This also makes me wonder, is there a linker that just supports all targets, ideally even across OSes? (Similar to how LLVM supports all architectures in a single binary.)

[bjorn3]

... This also makes me wonder, is there a linker that just supports all targets, ideally even across OSes? (Similar to how LLVM supports all architectures in a single binary.)

A bare linker that works across almost all targets that rustc supports exists. It is called lld. However on unix targets rustc doesn't invoke a bare linker, rather it invokes a linker driver like gcc or clang. This linker driver invokes the bare linker with a bunch of extra arguments specific to the target like where to find dynamic libraries (the sysroot), which crt.o files to link in and so on. While clang has most information baked in for all targets it supports, it doesn't and can't know where the sysroot is when cross-compiling. It could be located in a distro specific location or maybe the user wanted to install their own sysroot. For musl targets when link-self-contained is enabled, we should be able to get this working as we already ship all the necessary files, but for other targets shipping those files that is a lot harder as you have to deal with not every OS having syscall ABI stability or not allowing those files to be redistributed without license.

[RalfJung]

So this is more a reflection of the sorry state of cross-compilation on Unix in general? :(

However, having to install extra packages to support building other targets is not my main issue. What is really unfortunate is that installing the packages is not enough: even after installing them, an attempt at cross-compilation produces completely inscrutable error messages that are only fixed by setting two random config values (well, an environment variable and a cargo config) to tell them to actually use the thing I just installed. Is there ever a case where using cc for cross-compilation actually works? It seems to me the current default is pretty universally bad. OTOH it also seems like POSIX does not mandate there to be a single binary I can just invoke with particular flags to compile code for any target (any target that is supported by the current host, anyway), which is just sad.

Maybe we could at least detect the situation where a cross-build fails, and link to some documentation that explains what to do? Seems like cargo would be the place for that though, since it also has to catch build scripts failing.

I heard that Zig did some amazing things around cross-compilation. Could that be useful for us in any way?

[bjorn3]

One thing that would make cross-compilation easier is raw-dylib to avoid needing all dylibs being available at compile time. Windows supports it and #129821 will switch our Windows support to using it once it merges. There is some experimental support for ELF, but it currently doesn't support symbol versioning, which for example glibc needs.

[RalfJung]

Is the default of using cc for cross-compilation ever correct? Maybe instead of using that by default, cargo should just force some value to be set.

Also, the current state for build scripts just seems fundamentally broken. If a proc macro depends on a build script, it will need to use the host CC command, so setting just one CC for the entire build cannot be right. This also needs to be a per-target setting, similar to the linker.