Linking with `cargo-zigbuild` failed on `armv7-unknown-linux-{gnu, musl}eabihf` due to missing symbols

[NobodyXu]

release-builds / armv7-unknown-linux-musleabihf

release-builds / armv7-unknown-linux-gnueabihf

This can be reproduced by running:

git clone https://github.com/cargo-bins/cargo-binstall
cd cargo-binstall

export CI=true
export JUST_FOR_RELEASE=true
export CARGO_BUILD_TARGET="armv7-unknown-linux-gnueabihf"
export GLIBC_VERSION='2.17'
export JUST_USE_CARGO_ZIGBUILD=true
just build

Or by running the following cmds on the clone:

export RUSTFLAGS='-Z share-generics -C linker-plugin-lto -C link-arg=-static-libgcc -C link-arg=-Wl,--icf=safe'

cargo-zigbuild build  --release --target armv7-unknown-linux-musleabihf -Z build-std=std,panic_abort -Z build-std-features=panic_immediate_abort --config='profile.release.split-debuginfo="packed"' --config=profile.release.debug=2 --features static,rustls,trust-dns,fancy-no-backtrace,zstd-thin,log_release_max_level_debug,cross-lang-fat-lto

The CI failed with several missing symbols.

Also, cargo-zigbuild failed to handle several link args:

  = note: warning: unsupported linker arg: -znoexecstack
          warning: unsupported linker arg: -plugin-opt
          warning: unsupported linker arg: O2
          warning: unsupported linker arg: -plugin-opt
          warning: unsupported linker arg: mcpu=generic
          warning: unsupported linker arg: -zrelro
          warning: unsupported linker arg: -znow
          warning: unsupported linker arg: --icf
          warning: unsupported linker arg: safe

Version it worked on

It most recently worked on: rust version 1.69.0-nightly (7aa413d59 2023-02-19)

Version with regression

rust version 1.69.0-nightly (5243ea5c2 2023-02-20)

@rustbot modify labels: +regression-from-stable-to-nightly -regression-untriaged

[Urgau]

#108307 should have fixed the -z args. Could you try with the lastest nightly and report if anything has changed.

[NobodyXu]

#108307 should have fixed the -z args. Could you try with the lastest nightly and report if anything has changed.

@Urgau The CI still failed on 1.69.0-nightly (34e6673a0 2023-02-25).

[Urgau]

I see that the warnings for the unsupported -z args have now disappear (which is good), the other warnings are still there.

I also see that every linking errors have lto.tmp in them, which makes me thing that there might be an issue with LTO. Have to tried disabling it (to see if it fixes the problem) ?

---

Looking at the PR merged on the 2023-02-19 I don't see anything suspect. Could you try using cargo-rustc-bisect with a custom script to try to find the the commit that regressed your workflow.

[NobodyXu]

I also see that every linking errors have lto.tmp in them, which makes me thing that there might be an issue with LTO. Have to tried disabling it (to see if it fixes the problem) ?

I've tried disabling cross-lang-lto and feature "zstd/cross-lang-fat-lto".

It could be a lto problem, since the debug build, which does not turn on lto, works just fine.

Could you try using cargo-rustc-bisect with a custom script to try to find the the commit that regressed your workflow.

Yeah I could try this tomorrow.

[NobodyXu]

I'm using:

./test-regression.sh:

export CI=true
export JUST_FOR_RELEASE=true
export CARGO_BUILD_TARGET=armv7-unknown-linux-gnueabihf
export GLIBC_VERSION=2.17
export JUST_USE_CARGO_ZIGBUILD=true
just build

And then I am running cargo bisect-rustc --script ./test-regression.sh.
Will post the result tomorrow since it's too late now.

Edit:

It didn't work, will figure it out tomorrow.

[NobodyXu]

@Urgau Running:

export RUSTFLAGS='-Z share-generics -C linker-plugin-lto  -C link-arg=-Wl,--icf=safe'

cargo bisect-rustc \
    --by-commit --start '7aa413d59'\
    --target armv7-unknown-linux-gnueabihf --with-src --verbose -- \
    zigbuild --release --target armv7-unknown-linux-gnueabihf.2.17 \
    -Z build-std=std,panic_abort \
    -Z build-std-features=panic_immediate_abort \
    --config='profile.release.split-debuginfo="packed"' \
    --config=profile.release.debug=2 \
    --features static,rustls,trust-dns,fancy-no-backtrace,zstd-thin,log_release_max_level_debug,cross-lang-fat-lto

gives me:

checking the start range to verify it passes
installing 7aa413d59206fd511137728df3d9e0fd377429bd
rust-std-nightly-armv7-unknown-linux-gnueabihf: 26.55 MB / 26.55 MB [============] 100.00 % 949.12 KB/s 
testing...
RESULT: 7aa413d59206fd511137728df3d9e0fd377429bd, ===> Yes
uninstalling 7aa413d59206fd511137728df3d9e0fd377429bd

ERROR: the commit at the start of the range (7aa413d59206fd511137728df3d9e0fd377429bd) includes the regr
ession

which is strange, because this is the one that worked.

[Urgau]

The bisection wasn't successful. cargo-rustc-bisect needs to start from a known good commit/date. You need to choose a commit before this one, because 7aa413d5920 include the regression.

I would recommend using --start=2023-02-15 instead of --by-commit --start '7aa413d59'.

_{But I wouldn't be surprised if 7aa413d59 was the commit that regressed since it's the only one who modified the codegen on the 19 feb.}