redundant linker flag warning is wrong and breaks linking

[wthrowe]

If one passes the same -lfoo option twice to rustc it warns redundant linker flag specified for library `foo` and ignores the second argument. Linkers are sensitive to the order of their arguments and passing the same argument twice is common and sometimes necessary to get a program to link correctly.

In fact, rustc itself passes the linker (cleaned up a bit) -ldl -lrt -lpthread -lpthread -lgcc_s -lc -lm -lrt -lpthread -lutil -lutil, which includes four "redundant" linker flags (at least two of which are actually redundant, to be fair).

[Mark-Simulacrum]

This is somewhat not trivial to fix as we don't know whether a given -l flag was passed with the purpose of changing the link kind or name, or actually passing the same library twice for linkage. This branch has the initial workings of a viable idea, but it doesn't pass tests.

There are also to an extent some backwards compatibility concerns here.

---- [run-pass] run-pass/rfc1717/library-override.rs stdout ----

error: compilation failed!
status: exit code: 101
command: "/home/mark/Build/rust/build/x86_64-unknown-linux-gnu/stage1/bin/rustc" "/home/mark/Build/rust/src/test/run-pass/rfc1717/library-override.rs" "-L" "/home/mark/Build/rust/build/x86_64-unknown-linux-gnu/test/run-pass" "--target=x86_64-unknown-linux-gnu" "-C" "prefer-dynamic" "-o" "/home/mark/Build/rust/build/x86_64-unknown-linux-gnu/test/run-pass/rfc1717/library-override.stage1-x86_64-unknown-linux-gnu" "-Crpath" "-O" "-Zmiri" "-Zunstable-options" "-Lnative=/home/mark/Build/rust/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-lstatic=wronglibrary:rust_test_helpers" "-L" "/home/mark/Build/rust/build/x86_64-unknown-linux-gnu/test/run-pass/rfc1717/library-override.stage1-x86_64-unknown-linux-gnu.aux"
stdout:
------------------------------------------

------------------------------------------
stderr:
------------------------------------------
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/home/mark/Build/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/home/mark/Build/rust/build/x86_64-unknown-linux-gnu/test/run-pass/rfc1717/library-override.library_override0-317d481089b8c8fe83113de504472633.rs.rcgu.o" "/home/mark/Build/rust/build/x86_64-unknown-linux-gnu/test/run-pass/rfc1717/library-override.library_override1-317d481089b8c8fe83113de504472633.rs.rcgu.o" "-o" "/home/mark/Build/rust/build/x86_64-unknown-linux-gnu/test/run-pass/rfc1717/library-override.stage1-x86_64-unknown-linux-gnu" "/home/mark/Build/rust/build/x86_64-unknown-linux-gnu/test/run-pass/rfc1717/library-override.crate.allocator.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-nodefaultlibs" "-L" "/home/mark/Build/rust/build/x86_64-unknown-linux-gnu/test/run-pass" "-L" "/home/mark/Build/rust/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/home/mark/Build/rust/build/x86_64-unknown-linux-gnu/test/run-pass/rfc1717/library-override.stage1-x86_64-unknown-linux-gnu.aux" "-L" "/home/mark/Build/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "-Wl,--whole-archive" "-l" "rust_test_helpers" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "-l" "rust_test_helpers" "-Wl,--no-whole-archive" "-L" "/home/mark/Build/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bdynamic" "-l" "std-57fdceb1c52b4db2" "-Wl,-Bstatic" "/home/mark/Build/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-136e26942e0df602.rlib" "-Wl,-Bdynamic" "-l" "dl" "-l" "rt" "-l" "pthread" "-l" "pthread" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "pthread" "-l" "util" "-l" "util" "-Wl,-rpath,$ORIGIN/../../../stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-rpath,/home/mark/Build/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,--enable-new-dtags"
  = note: /home/mark/Build/rust/build/x86_64-unknown-linux-gnu/native/rust-test-helpers/librust_test_helpers.a(rust_test_helpers.o): In function `rust_dbg_extern_identity_u32':
          rust_test_helpers.c:(.text.rust_dbg_extern_identity_u32+0x0): multiple definition of `rust_dbg_extern_identity_u32'
          /home/mark/Build/rust/build/x86_64-unknown-linux-gnu/native/rust-test-helpers/librust_test_helpers.a(rust_test_helpers.o):rust_test_helpers.c:(.text.rust_dbg_extern_identity_u32+0x0): first defined here
          /home/mark/Build/rust/build/x86_64-unknown-linux-gnu/native/rust-test-helpers/librust_test_helpers.a(rust_test_helpers.o): In function `rust_dbg_extern_identity_u64':
          rust_test_helpers.c:(.text.rust_dbg_extern_identity_u64+0x0): multiple definition of `rust_dbg_extern_identity_u64'
          /home/mark/Build/rust/build/x86_64-unknown-linux-gnu/native/rust-test-helpers/librust_test_helpers.a(rust_test_helpers.o):rust_test_helpers.c:(.text.rust_dbg_extern_identity_u64+0x0): first defined here
...

[lovesegfault]

Could we perhaps at least add some option to ignore this?

[dcreager]

I was recently bitten by this bug, trying to link multiple internal helper C++ libraries into a Rust executable. I was able to put together a minimal repro: on Linux, one of the subpackages builds successfully, and one gets linker errors, and the only difference between the two is the order in which the helper libraries are built and added to the linker command line.

[dcreager]

I wrote up a potential fix in #57018. We'd still remove redundant -l flags from the linker command, but instead of keeping the first one, we'd now keep the last one. The new behavior passes CI, and I've verified (via cargo +local build) that it also fixes all of the linker errors in my repro repo.