libffi build error on aarch64-gnu-debug: unsupported option '-print-multi-os-directory'

[RalfJung]

This seems to happen every now and then, see the search results, e.g. #147551, #147444, #147468

https://triage.rust-lang.org/gha-logs/rust-lang/rust/52441546475

2025-10-10T12:07:12.7435770Z warning: libffi-sys@3.3.2: Configuring libffi failed with exit status: 1
2025-10-10T12:07:12.7437253Z error: failed to run custom build command for `libffi-sys v3.3.2`
2025-10-10T12:07:12.7438077Z 
2025-10-10T12:07:12.7438302Z Caused by:
2025-10-10T12:07:12.7441406Z   process didn't exit successfully: `/checkout/obj/build/aarch64-unknown-linux-gnu/stage2-tools/release/build/libffi-sys-a7c6d49d23c92c87/build-script-build` (exit status: 101)
2025-10-10T12:07:12.7442186Z   --- stdout
2025-10-10T12:07:12.7442596Z   checking build system type... aarch64-unknown-linux-gnu
2025-10-10T12:07:12.7442949Z   checking host system type... aarch64-unknown-linux-gnu
2025-10-10T12:07:12.7443418Z   checking target system type... aarch64-unknown-linux-gnu
2025-10-10T12:07:12.7444031Z   continue configure in default builddir "./aarch64-unknown-linux-gnu"
2025-10-10T12:07:12.7444972Z   ....exec /bin/bash .././configure "--srcdir=.." "--enable-builddir=aarch64-unknown-linux-gnu" "linux
2025-10-10T12:07:12.7445802Z   gnu"
2025-10-10T12:07:12.7446346Z   checking build system type... aarch64-unknown-linux-gnu
2025-10-10T12:07:12.7446888Z   checking host system type... aarch64-unknown-linux-gnu
2025-10-10T12:07:12.7447526Z   checking target system type... aarch64-unknown-linux-gnu
2025-10-10T12:07:12.7448049Z   checking for gsed... sed
2025-10-10T12:07:12.7448627Z   checking for a BSD-compatible install... /usr/bin/install -c
2025-10-10T12:07:12.7449172Z   checking whether build environment is sane... yes
2025-10-10T12:07:12.7449802Z   checking for a race-free mkdir -p... ../install-sh -c -d
2025-10-10T12:07:12.7450400Z   checking for gawk... no
2025-10-10T12:07:12.7450767Z   checking for mawk... mawk
2025-10-10T12:07:12.7451311Z   checking whether make sets $(MAKE)... yes
2025-10-10T12:07:12.7452158Z   checking whether make supports nested variables... yes
2025-10-10T12:07:12.7452814Z   checking for gcc... clang
2025-10-10T12:07:12.7453697Z   checking whether the C compiler works... yes
2025-10-10T12:07:12.7454426Z   checking for C compiler default output file name... a.out
2025-10-10T12:07:12.7455118Z   checking for suffix of executables... 
2025-10-10T12:07:12.7455828Z   checking whether we are cross compiling... no
2025-10-10T12:07:12.7456164Z   checking for suffix of object files... o
2025-10-10T12:07:12.7456471Z   checking whether the compiler supports GNU C... yes
2025-10-10T12:07:12.7456777Z   checking whether clang accepts -g... yes
2025-10-10T12:07:12.7457104Z   checking for clang option to enable C11 features... none needed
2025-10-10T12:07:12.7457486Z   checking whether clang understands -c and -o together... yes
2025-10-10T12:07:12.7457897Z   checking whether make supports the include directive... yes (GNU style)
2025-10-10T12:07:12.7458261Z   checking dependency style of clang... gcc3
2025-10-10T12:07:12.7458535Z   checking for g++... g++
2025-10-10T12:07:12.7458794Z   checking whether the compiler supports GNU C++... yes
2025-10-10T12:07:12.7459102Z   checking whether g++ accepts -g... yes
2025-10-10T12:07:12.7459420Z   checking for g++ option to enable C++11 features... none needed
2025-10-10T12:07:12.7459743Z   checking dependency style of g++... gcc3
2025-10-10T12:07:12.7460010Z   checking dependency style of clang... gcc3
2025-10-10T12:07:12.7460335Z   checking for grep that handles long lines and -e... /usr/bin/grep
2025-10-10T12:07:12.7460662Z   checking for egrep... /usr/bin/grep -E
2025-10-10T12:07:12.7460927Z   checking how to print strings... printf
2025-10-10T12:07:12.7461249Z   checking for a sed that does not truncate output... /usr/bin/sed
2025-10-10T12:07:12.7461570Z   checking for fgrep... /usr/bin/grep -F
2025-10-10T12:07:12.7461838Z   checking for ld used by clang... /usr/bin/ld
2025-10-10T12:07:12.7462141Z   checking if the linker (/usr/bin/ld) is GNU ld... yes
2025-10-10T12:07:12.7462517Z   checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
2025-10-10T12:07:12.7462917Z   checking the name lister (/usr/bin/nm -B) interface... BSD nm
2025-10-10T12:07:12.7463233Z   checking whether ln -s works... yes
2025-10-10T12:07:12.7463549Z   checking the maximum length of command line arguments... 3145728
2025-10-10T12:07:12.7464115Z   checking how to convert aarch64-unknown-linux-gnu file names to aarch64-unknown-linux-gnu format... func_convert_file_noop
2025-10-10T12:07:12.7465001Z   checking how to convert aarch64-unknown-linux-gnu file names to toolchain format... func_convert_file_noop
2025-10-10T12:07:12.7465571Z   checking for /usr/bin/ld option to reload object files... -r
2025-10-10T12:07:12.7465876Z   checking for file... file
2025-10-10T12:07:12.7466098Z   checking for objdump... objdump
2025-10-10T12:07:12.7466391Z   checking how to recognize dependent libraries... pass_all
2025-10-10T12:07:12.7466691Z   checking for dlltool... no
2025-10-10T12:07:12.7467000Z   checking how to associate runtime and link libraries... printf %s\n
2025-10-10T12:07:12.7467319Z   checking for ar... ar
2025-10-10T12:07:12.7467546Z   checking for archiver @FILE support... @
2025-10-10T12:07:12.7467805Z   checking for strip... strip
2025-10-10T12:07:12.7468029Z   checking for ranlib... ranlib
2025-10-10T12:07:12.7468348Z   checking command to parse /usr/bin/nm -B output from clang object... ok
2025-10-10T12:07:12.7468685Z   checking for sysroot... no
2025-10-10T12:07:12.7468918Z   checking for a working dd... /usr/bin/dd
2025-10-10T12:07:12.7469246Z   checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1
2025-10-10T12:07:12.7469563Z   checking for mt... no
2025-10-10T12:07:12.7469781Z   checking if : is a manifest tool... no
2025-10-10T12:07:12.7470032Z   checking for stdio.h... yes
2025-10-10T12:07:12.7470251Z   checking for stdlib.h... yes
2025-10-10T12:07:12.7470478Z   checking for string.h... yes
2025-10-10T12:07:12.7470710Z   checking for inttypes.h... yes
2025-10-10T12:07:12.7471324Z   checking for stdint.h... yes
2025-10-10T12:07:12.7471623Z   checking for strings.h... yes
2025-10-10T12:07:12.7471978Z   checking for sys/stat.h... yes
2025-10-10T12:07:12.7472212Z   checking for sys/types.h... yes
2025-10-10T12:07:12.7472446Z   checking for unistd.h... yes
2025-10-10T12:07:12.7472670Z   checking for dlfcn.h... yes
2025-10-10T12:07:12.7472891Z   checking for objdir... .libs
2025-10-10T12:07:12.7473178Z   checking if clang supports -fno-rtti -fno-exceptions... yes
2025-10-10T12:07:12.7473540Z   checking for clang option to produce PIC... -fPIC -DPIC
2025-10-10T12:07:12.7473874Z   checking if clang PIC flag -fPIC -DPIC works... yes
2025-10-10T12:07:12.7474195Z   checking if clang static flag -static works... yes
2025-10-10T12:07:12.7474555Z   checking if clang supports -c -o file.o... yes
2025-10-10T12:07:12.7475062Z   checking if clang supports -c -o file.o... (cached) yes
2025-10-10T12:07:12.7475473Z   checking whether the clang linker (/usr/bin/ld) supports shared libraries... yes
2025-10-10T12:07:12.7475904Z   checking dynamic linker characteristics... GNU/Linux ld.so
2025-10-10T12:07:12.7476287Z   checking how to hardcode library paths into programs... immediate
2025-10-10T12:07:12.7476661Z   checking whether stripping libraries is possible... yes
2025-10-10T12:07:12.7477001Z   checking if libtool supports shared libraries... yes
2025-10-10T12:07:12.7477314Z   checking whether to build shared libraries... no
2025-10-10T12:07:12.7477621Z   checking whether to build static libraries... yes
2025-10-10T12:07:12.7477937Z   checking how to run the C++ preprocessor... g++ -E
2025-10-10T12:07:12.7478229Z   checking for ld used by g++... /usr/bin/ld
2025-10-10T12:07:12.7478520Z   checking if the linker (/usr/bin/ld) is GNU ld... yes
2025-10-10T12:07:12.7478906Z   checking whether the g++ linker (/usr/bin/ld) supports shared libraries... yes
2025-10-10T12:07:12.7479304Z   checking for g++ option to produce PIC... -fPIC -DPIC
2025-10-10T12:07:12.7479625Z   checking if g++ PIC flag -fPIC -DPIC works... yes
2025-10-10T12:07:12.7479932Z   checking if g++ static flag -static works... yes
2025-10-10T12:07:12.7480233Z   checking if g++ supports -c -o file.o... yes
2025-10-10T12:07:12.7480536Z   checking if g++ supports -c -o file.o... (cached) yes
2025-10-10T12:07:12.7480921Z   checking whether the g++ linker (/usr/bin/ld) supports shared libraries... yes
2025-10-10T12:07:12.7481358Z   checking dynamic linker characteristics... (cached) GNU/Linux ld.so
2025-10-10T12:07:12.7481764Z   checking how to hardcode library paths into programs... immediate
2025-10-10T12:07:12.7482088Z   checking for readelf... readelf
2025-10-10T12:07:12.7482325Z   checking size of size_t... 8
2025-10-10T12:07:12.7482563Z   checking for C compiler vendor... clang
2025-10-10T12:07:12.7482863Z   checking CFLAGS for most reasonable warnings... -Wall
2025-10-10T12:07:12.7483262Z   checking whether to enable maintainer-specific portions of Makefiles... no
2025-10-10T12:07:12.7483620Z   checking for sys/memfd.h... no
2025-10-10T12:07:12.7483864Z   checking for memfd_create... yes
2025-10-10T12:07:12.7484128Z   checking for egrep... (cached) /usr/bin/grep -E
2025-10-10T12:07:12.7484410Z   checking for memcpy... yes
2025-10-10T12:07:12.7484828Z   checking for alloca.h... yes
2025-10-10T12:07:12.7485084Z   checking size of double... 8
2025-10-10T12:07:12.7485316Z   checking size of long double... 16
2025-10-10T12:07:12.7485592Z   checking whether byte ordering is bigendian... no
2025-10-10T12:07:12.7485905Z   checking assembler .cfi pseudo-op support... yes
2025-10-10T12:07:12.7486264Z   checking whether compiler supports pointer authentication... no
2025-10-10T12:07:12.7486615Z   checking for _ prefix in compiled symbols... no
2025-10-10T12:07:12.7486928Z   checking whether C compiler accepts -fno-lto... yes
2025-10-10T12:07:12.7487280Z   checking whether .eh_frame section should be read-only... yes
2025-10-10T12:07:12.7487643Z   checking for __attribute__((visibility("hidden")))... yes
2025-10-10T12:07:12.7488275Z   configure: versioning on shared library symbols is no
2025-10-10T12:07:12.7488651Z   checking that generated files are newer than configure... done
2025-10-10T12:07:12.7489092Z   configure: creating ./config.status
2025-10-10T12:07:12.7489356Z   config.status: creating include/Makefile
2025-10-10T12:07:12.7489620Z   config.status: creating include/ffi.h
2025-10-10T12:07:12.7489874Z   config.status: creating Makefile
2025-10-10T12:07:12.7490127Z   config.status: creating testsuite/Makefile
2025-10-10T12:07:12.7490397Z   config.status: creating man/Makefile
2025-10-10T12:07:12.7490650Z   config.status: creating doc/Makefile
2025-10-10T12:07:12.7490928Z   config.status: creating libffi.pc
2025-10-10T12:07:12.7491176Z   config.status: creating fficonfig.h
2025-10-10T12:07:12.7491434Z   config.status: executing buildir commands
2025-10-10T12:07:12.7491775Z   config.status: create top_srcdir/Makefile guessed from local Makefile
2025-10-10T12:07:12.7492242Z   config.status: build in aarch64-unknown-linux-gnu (HOST=aarch64-unknown-linux-gnu)
2025-10-10T12:07:12.7492647Z   config.status: executing depfiles commands
2025-10-10T12:07:12.7492977Z   cargo:warning=Configuring libffi failed with exit status: 1
2025-10-10T12:07:12.7493210Z 
2025-10-10T12:07:12.7493280Z   --- stderr
2025-10-10T12:07:12.7493531Z   clang: error: unsupported option '-print-multi-os-directory'
2025-10-10T12:07:12.7493844Z   clang: error: no input files
2025-10-10T12:07:12.7494601Z   config.status: error: in `/checkout/obj/build/aarch64-unknown-linux-gnu/stage2-tools/aarch64-unknown-linux-gnu/release/build/libffi-sys-6562d98fd0e95724/out/libffi-build/aarch64-unknown-linux-gnu':
2025-10-10T12:07:12.7495691Z   config.status: error: Something went wrong bootstrapping makefile fragments
2025-10-10T12:07:12.7496148Z       for automatic dependency tracking.  If GNU make was not used, consider
2025-10-10T12:07:12.7496565Z       re-running the configure script with MAKE="gmake" (or whatever is
2025-10-10T12:07:12.7496946Z       necessary).  You can also try re-running configure with the
2025-10-10T12:07:12.7497350Z       '--disable-dependency-tracking' option to at least be able to build
2025-10-10T12:07:12.7497791Z       the package (albeit without support for automatic dependency tracking).
2025-10-10T12:07:12.7498137Z   See `config.log' for more details
2025-10-10T12:07:12.7498296Z 
2025-10-10T12:07:12.7498648Z   thread 'main' (98754) panicked at /cargo/registry/src/index.crates.io-1949cf8c6b5b557f/libffi-sys-3.3.2/build/not_msvc.rs:154:5:
2025-10-10T12:07:12.7500886Z   Configuring libffi: exit status: 1 (cd "/checkout/obj/build/aarch64-unknown-linux-gnu/stage2-tools/aarch64-unknown-linux-gnu/release/build/libffi-sys-6562d98fd0e95724/out/libffi-build" && CC="clang" CFLAGS="-O3 -ffunction-sections -fdata-sections -fPIC -gdwarf-4 -fno-omit-frame-pointer --target=aarch64-unknown-linux-gnu -Wno-implicit-function-declaration" LC_ALL="C" "sh" "./configure" "--with-pic" "--disable-shared" "--disable-docs" "--prefix" "/checkout/obj/build/aarch64-unknown-linux-gnu/stage2-tools/aarch64-unknown-linux-gnu/release/build/libffi-sys-6562d98fd0e95724/out/libffi-root")

I have no idea how this issue can occur spuriously...

[bjorn3]

Clang doesn't support that option: https://github.com/llvm/llvm-project/blob/1d3ad74667e86a1ba19285e58d14156cce3de89d/clang/include/clang/Driver/Options.td#L6061-L6062 And it has been producing an error since 2015: llvm/llvm-project@3d5b610 libffi is supposed to not pass it when the compiler is not GCC: https://github.com/libffi/libffi/blob/2835f72cc7ee57edfc987da4b88b1f4c7c0386c3/configure.ac#L415 I have no clue how the GCC detection works.

[jieyouxu]

Observed on aarch64-gnu likewise in #147498 (comment).

[RalfJung]

Observed on aarch64-gnu likewise in #147498 (comment).

Is that the same error? It doesn't have the print-multi-os-directory part.

[RalfJung]

libffi-sys hasn't changed since 31ea5f9. So something else must have changed that makes this happen now...

[jieyouxu]

Is that the same error? It doesn't have the print-multi-os-directory part.

Hm good point... that error doesn't actually tell us explicitly what failed 🤔

[weihanglo]

Also #147568 (comment) and #147574 (comment). Should we do a treeclosed?

[jieyouxu]

Yeah, looks like we probably need to. I'll close tree for now, if the cargo update PR somehow manages to get past this failure, then I'll reopen (though it's still way too flaky if this is not always failing).

[ehuss]

From what I can tell, the Ubuntu 25.10 image was updated in the past 24 hours. I don't see a way on dockerhub to see what changed. I see docker-library/repo-info@9a7f0d7#diff-adfeb36a04a9eb932b6b7fb81b0664c017f53abacad4782d5c5111edee9712e9 which looks like a bunch of stuff was updated.

[jieyouxu]

cc infra zulip thread: #t-infra > Treeclosed: libffi failures