[libunwind] Wrong default for LIBUNWIND_ENABLE_FRAME_APIS?

[xarblu]

In short - I recently hit the following issue with >=libunwind-18.1.0_rc1 on Gentoo:
Anything using libunwind started failing with /usr/lib64/libc++abi.so.1: undefined symbol: __deregister_frame_info.
For more info the downstream bugreport is here.

Eventually I tracked it down to the newly introduced LIBUNWIND_ENABLE_FRAME_APIS defaulting to OFF (5eb44df).

What confuses me is that the commit message clearly says:

Default this to on, so as not to remove the apis from
environments that already have them.

but the actual option defaults to OFF.

(Downstream said to CC author @Sterling-Augustine and reviewer @MaskRay)

[Sterling-Augustine]

I apologize for not updating the commit message to reflect the final decision, which went through a bunch of revisions, and I missed that fix. I don't know why libc++abi.so.1 is referencing __deregister_frame_info. It really shouldn't be. The function in llvm libunwind never did anything and has always been an empty stub. Even worse, it's presence has always been guarded by an erroneous #ifdef--based on target architecture, not target OS. So sometimes it was present and sometimes it wasn't. This build likely wouldn't have worked on some of the other target architectures even before this change. Any reference to this function is almost surely a bug. I can't see any direct reference to it from libc++, so somehow the build has included it. A wild guess is that the build is including EH_USE_FRAME_REGISTRY, which I'm not sure why one would use that these days.

…

On Mon, Feb 26, 2024 at 3:25 PM Xarblu ***@***.***> wrote: In short - I recently hit the following issue with >=libunwind-18.1.0_rc1 on Gentoo: *Anything* using libunwind started failing with /usr/lib64/libc++abi.so.1: undefined symbol: __deregister_frame_info. For more info the downstream bugreport is here <https://bugs.gentoo.org/925024>. Eventually I tracked it down to the newly introduced LIBUNWIND_ENABLE_FRAME_APIS defaulting to OFF (5eb44df <5eb44df> ). What confuses me is that the commit message clearly says: Default this to on, so as not to remove the apis from environments that already have them. but the actual option defaults to OFF. (Downstream said to CC author @Sterling-Augustine <https://github.com/Sterling-Augustine> and reviewer @MaskRay <https://github.com/MaskRay>) — Reply to this email directly, view it on GitHub <#83080>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANSXMSW7C3SSGFQ52TWB63LYVUKXTAVCNFSM6AAAAABD3CZLMCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGE2TKMZUHEYDKMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Sterling-Augustine]

And by "the build", I mean the build of libc++.so On Mon, Feb 26, 2024 at 3:41 PM Sterling Augustine ***@***.***> wrote:

…

I apologize for not updating the commit message to reflect the final decision, which went through a bunch of revisions, and I missed that fix. I don't know why libc++abi.so.1 is referencing __deregister_frame_info. It really shouldn't be. The function in llvm libunwind never did anything and has always been an empty stub. Even worse, it's presence has always been guarded by an erroneous #ifdef--based on target architecture, not target OS. So sometimes it was present and sometimes it wasn't. This build likely wouldn't have worked on some of the other target architectures even before this change. Any reference to this function is almost surely a bug. I can't see any direct reference to it from libc++, so somehow the build has included it. A wild guess is that the build is including EH_USE_FRAME_REGISTRY, which I'm not sure why one would use that these days. On Mon, Feb 26, 2024 at 3:25 PM Xarblu ***@***.***> wrote: > In short - I recently hit the following issue with >=libunwind-18.1.0_rc1 > on Gentoo: > *Anything* using libunwind started failing with /usr/lib64/libc++abi.so.1: > undefined symbol: __deregister_frame_info. > For more info the downstream bugreport is here > <https://bugs.gentoo.org/925024>. > > Eventually I tracked it down to the newly introduced > LIBUNWIND_ENABLE_FRAME_APIS defaulting to OFF (5eb44df > <5eb44df> > ). > > What confuses me is that the commit message clearly says: > > Default this to on, so as not to remove the apis from > environments that already have them. > > but the actual option defaults to OFF. > > (Downstream said to CC author @Sterling-Augustine > <https://github.com/Sterling-Augustine> and reviewer @MaskRay > <https://github.com/MaskRay>) > > — > Reply to this email directly, view it on GitHub > <#83080>, or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ANSXMSW7C3SSGFQ52TWB63LYVUKXTAVCNFSM6AAAAABD3CZLMCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGE2TKMZUHEYDKMQ> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[xarblu]

A wild guess is that the build is including
EH_USE_FRAME_REGISTRY, which I'm not sure why one would use that these days.

Could this come from COMPILER_RT_CRT_USE_EH_FRAME_REGISTRY defaulting to ON? (seen here)

[MaskRay]

symbol lookup error: indicates a runtime glibc error likely when glibc fails to find a definition for a relocation referencing __deregister_frame_info. clang_rt.crtbegin.o from compiler-rt does reference __deregister_frame_info but it's weak, allowed to be unresolved. Therefore, I am puzzled how you get the error from /usr/lib64/libc++abi.so.1. Run a readelf -Ws on it and rerun the libc++abi.so.1 link step with -Wl,-y,__deregister_frame_info and dump the relevant files?

I agree that we can probably make EH_USE_FRAME_REGISTRY default to off.

Clang and GCC usually pass --eh-frame-hdr to ld, with the exception that gcc -static does not pass --eh-frame-hdr. The difference is a historical choice related to __register_frame_info.

[MaskRay]

symbol lookup error: indicates a runtime glibc error likely when glibc fails to find a definition for a relocation referencing __deregister_frame_info. clang_rt.crtbegin.o from compiler-rt does reference __deregister_frame_info but it's weak, allowed to be unresolved. Therefore, I am puzzled how you get the error from /usr/lib64/libc++abi.so.1. Run a readelf -Ws on it and rerun the libc++abi.so.1 link step with -Wl,-y,__deregister_frame_info and dump the relevant files?

I agree that we can probably make EH_USE_FRAME_REGISTRY default to off.

Clang and GCC usually pass --eh-frame-hdr to ld, with the exception that gcc -static does not pass --eh-frame-hdr. The difference is a historical choice related to __register_frame_info.

Created #83201 to default to COMPILER_RT_CRT_USE_EH_FRAME_REGISTRY=off

[xarblu]

Run a readelf -Ws on it and rerun the libc++abi.so.1 link step with -Wl,-y,__deregister_frame_info and dump the relevant files?

$ readelf -Ws --demangle /usr/lib64/libc++abi.so.1.0 | grep frame_info
1: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND __deregister_frame_info
3: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND __register_frame_info

Linking with Wl,-y,__deregister_frame_info mentions:

/usr/lib/llvm/18/bin/../../../../lib/clang/18/lib/linux/clang_rt.crtbegin-x86_64.o: reference to __deregister_frame_info
/usr/lib/gcc/x86_64-pc-linux-gnu/13/../../../../lib64/libunwind.so: shared definition of __deregister_frame_info

And readelf -Ws --demangle on those:

# /usr/lib/llvm/18/bin/../../../../lib/clang/18/lib/linux/clang_rt.crtbegin-x86_64.o
# (aka /usr/lib/clang/18/lib/linux/clang_rt.crtbegin-x86_64.o)
4: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __register_frame_info
7: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __deregister_frame_info

# /usr/lib/gcc/x86_64-pc-linux-gnu/13/../../../../lib64/libunwind.so
# (aka /usr/lib64/libunwind.so.1.0, llvm-libunwind, built with default -DLIBUNWIND_ENABLE_FRAME_APIS=OFF)
1: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND __deregister_frame_info
3: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND __register_frame_info

They all have bind: WEAK so they should theoretically be fine with the symbols missing, no?

However I think I might've found the real culprit here: ld.mold
readelfing various libraries revealed that pretty much everything linked by ld.mold has bind: GLOBAL for __deregister_frame_info (which is now unavailable unless -DLIBUNWIND_ENABLE_FRAME_APIS=ON).
Linking with ld.lld gives the expected bind: WEAK.
(Just rebuilt libcxx(abi) with lld while writing and now I get clang: symbol lookup error: /usr/lib64/libicuuc.so.74: undefined symbol: __deregister_frame_info. So yeah, it's very likely a mold issue.)

[xarblu]

Finished rebuilding various libraries that made clang --version complain about the undefined symbol with lld until it worked. (They all went from bind: GLOBAL -> WEAK)
That pretty much confirms for me that it was mold from the start which only got revealed when llvm-libunwind dropped the symbols.

Sorry for the noise but thanks for helping me figure this out even if it ended up not being an LLVM issue.

I guess that means this can be closed (and moved to the mold folks)? Unless there's still something for the LLVM side?

[thesamesam]

cc @rui314