stop exporting every symbol

[m4b]

On Linux, a typical rust ELF dynamic library will export every symbol on earth, e.g.:

objdump -T ~/projects/goblin/target/debug/libgoblin.so | wc -l
2558

which looks like most of core and std, e.g.:

107680 _ZN4core3num52_$LT$impl$u20$core..str..FromStr$u20$for$u20$u16$GT$8from_str17he60266b17184e647E (114)

Why not do this? Well, it's just The Right Thing To Do ™️ but also, technically it unnecessarily increases the size of the binary by adding dynamic symbol entries + string table entires, in addition to bloom filter words.

It might slightly increase dynamic loading time but I'm probably lying now to increase bullet points for why we shouldn't export all the symbols.

I suspect this may have come up already but I can't seem to find it, but I recall perhaps it had something to do with a requirement of std to make all their symbols public and this is somehow reflected in the binary? or perhaps it was something to do with more 3rd party linker nonsense.

Anyway, if there's a technical possibility to stop it, we should try and be good binary citizens.

[retep998]

If libgoblib.so is a dylib and not a cdylib, and you statically linked in std and friends, this means that not only is it intended to be consumed by other Rust crates, but it is also responsible for providing all of std and friends. If you don't want it providing std and friends, then you should use -Cprefer-dynamic so they're pulled in as dylib dependencies too. If you want to create something to be consumed by C, and not by Rust, then use cdylib instead which strips out all the bloat entirely.

[m4b]

@retep998 Adding cdylib seems to have no effect on my linux system (although admittedly I had no idea this distinction was made :] )

I thought perhaps it needed at least an unmangled symbol to "kick" in, but this doesn't seem to help either.

objdump -T target/debug/libdylib.so  | wc -l
2250

cat Cargo.toml 
[package]
name = "dylib"
version = "0.1.0"
authors = ["m4b <m4b.github.io@gmail.com>"]

[lib]
crate-type = ["cdylib"]

[dependencies]

cat src/lib.rs 
#[no_mangle]
pub extern fn test () {
    println!("yup");
}

[alexcrichton]

@m4b that's likely a bug in cdylibs if it's doing the wrong thing here. The intention of cdylibs are to only export a C interface, which in the past I've inspected via nm -g (which for the cdylib above has very few symbols exported). I don't know the relationship between nm -g and objdump -T, however.

[m4b]

wasn't aware of nm -g, but it seems to just be filtering all symbols in the symbol table if they are GLOBAL and FUNC.

objdump -T prints every symbol in the dynamic table entry (seems equivalent to nm -D).

nm -g unfortunately isn't sufficient, and it's really kind of lying about what the "exported" symbols are (i.e., the things other libraries can import into themselves). To test this, run strip target/debug/libdylib.so and then nm -g; it will report no symbols, because the symbol table (which is strippable) was removed. Stripping released binaries on distros is fairly common (unfortunatley, imho), as it reduces network transfer size and isn't required for dynamic linking.

This does not affect the importability of test, which is given by the symbol table referenced in the so-called _DYNAMIC array in a binary (which is what the dynamic linker accesses to see if it has a symbol another binary is referencing).

[alexcrichton]

This may be related to #32887 as well. I don't really understand most of the terms being mentioned there and how they relate to each other personally, but patches are of course always welcome!

[m4b]

I think that is related to another issue I saw with rust internal functions being assigned JUMP_SLOT relocations, which is very unusual, but didn't raise an issue (yet). I will read the above issue more thoroughly to see if it's related, but I think your suspicion that they're related might be correct. If the internal functions are given PLT entries, (and hence the JUMP_SLOT relocations), this would probably qualifiy them to be pushed into the dynamic, exported symbols array by the static linker :/

[m4b]

Motivation

So. I think we can fix this (and it is something that eventually needs fixing). To illustrate what I think we're on target for, the snappy shared object library is a good comparison because:

1. it exposes a C ABI
2. it exposes it's native language ABI (C++)

The resulting ELF ABI exactly parallels the two API headers, i.e.:

nm -D /usr/lib/libsnappy.so
0000000000005ac0 T snappy_compress
0000000000005ab0 T snappy_max_compressed_length
0000000000005a10 T snappy_uncompress
0000000000005b20 T snappy_uncompressed_length
0000000000005b40 T snappy_validate_compressed_buffer
                 U __stack_chk_fail
                 U _Unwind_Resume
                 U _ZdaPv
                 U _ZdlPv
0000000000004290 T _ZN6snappy10UncompressEPKcmPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
0000000000004990 T _ZN6snappy10UncompressEPNS_6SourceEPNS_4SinkE
0000000000004410 T _ZN6snappy11RawCompressEPKcmPcPm
0000000000004210 T _ZN6snappy13RawUncompressEPKcmPc
0000000000003d90 T _ZN6snappy13RawUncompressEPNS_6SourceEPc
...

On my distro, this binary is stripped, and nm -D libsnappy.so | wc -l is 90. If the shared object had re-exported the parts of libstdc++.so it used, it would be huge (which is what we're doing). This binary allows dynamic linking against public symbols in both it's C api, and it's C++ api, without adding a ton of symbols unrelated to it's api, which of course, are public, otherwise it couldn't link/dynamically link against them. (an example of this without an exposed C++ api is libz.so, which similarly is stripped but only exports about 40 symbols in it's dynamic symbol table).

Algo

Anyway, I've found a few places for attack, potential candidates being mostly in rustc_trans/base.rs, maybe starting around here or perhaps here.

So I'm not really familiar with the rust compiler internals at all and all the ctx and sess's and ttx's are terrifying me right now, but I think the basic algorithm is as follows:

if the crate is the base crate (i.e., the final artifact we're compiling) && it is a cdylib then:
for every symbol in final artifact:
if symbol visibility not pub && not is_extern && symbol is not defined in base crate:
llvm::LLVMSetVisibility(val, Hidden)
(or perhaps set the linkage to private, need to test)

I don't know enough about rustc internals to easily write "is the base crate" (i assume it's crate num zero, but I don't see it documented anywhere), or whether currently there is such a distinction.

I also don't see an easy way to write "is defined in base crate", as the first link I gave just ends up slurping all the symbols into a vector of reachable symbols, which is just an array of strs, but I'm sure it's somewhere; probably in one of those context references ;)

The algo might seem a little strange, but modulo me messing up booleans (which is likely), I think it's correct. Exposing a C API should not re-export all public or even extern symbols from either dynamic or statically linked dependencies. It should only export public extern definitions which we ourselves have created; hence, only public, extern symbols defined in our crate/lib. This is what libsnappy.so, libz.so and any other shared object distributed and compiled with a "normal" compiler for a C/C++ language will typically do.

The only other thing to watch out for will be the prefer-dynamic flag, which means any pub symbol from an external crate (i.e., one we didn't define), must be placed in the dynamic array (and hence be imported). The hidden flag will prevent this iirc, and we might need to instead adjust the linkage to available_externally, again, dunno, need to test.

(also, on cursory examination, this appears to not be working either? passing cargo rustc -- -C prefer-dynamic to the test crate above does not yield a binary with dynamic rust library dependencies, which I thought was it's purpose).

Anyway, yea I think it's doable, with some kind of riffing on the above algorithm.

P.S. I suspect this will remove every single one of those JUMP_SLOT relocations for the pub symbols coming from dependencies which are statically linked in, since they're now hidden, not in the dynamic array (or any symbol table), and so the linker should be free to optimize.

[michaelwoerister]

cc me

[alexcrichton]

@m4b AFAIK yeah dealing with symbol visibility (particularly hidden) is the way to tackle this. The only real hard case I think is what to do when one of the output artifacts is an rlib. If all we're producing is a cdylib, dylib, executable, or staticlib, then we can make as many symbols hidden as possible as they're all private impl details.

If we're producing an rlib, however, then this rlib will become one of a few things:

• A private implementation detail of another staticlib/dylib/executable
• A public dependency of a dylib
• A public dependency with a stable ABI (e.g. an extern interface) of a staticlib/dylib/executable

We unfortunately don't know which, we we'd have to pessimistically assume that it could become a public dependency of a dylib, causing the symbols to not be hidden...

I may not be understanding what hidden means though. We should really just play around with the options here and see what happens. If the bootstrap and tests succeed then whatever system we have should work fine.

[michaelwoerister]

FYI: I've implemented what is probably a small improvement in the right direction:
https://github.com/michaelwoerister/rust/tree/symbol-visibility

It makes sure that symbols get hidden if the only reason they are externally visible was that they are shared between object files of the same crate. You can give it a try if you build that branch.

I haven't really thought hard about all of this though. One thing to note though: We might be switching to MIR-only rlibs at some point, which would make all of this a lot easier, I suspect.

[m4b]

@michaelwoerister Your patch looks like a good start!

Unfortunately it does not seem to affect symbol visibility?

I compiled several binaries with your changes, and the only difference I noticed was that rust static libs had PC32 relocations instead PLT32 irrc.

However, strangely enough, I noticed when passing the -lto flag to compile cdylib (which doesn't seem like it should be allowed according to usage error if you try with regular dylib: error: lto can only be run for executables and static library outputs) the resulting binary with -lto is almost exactly what we want, e.g., lto and non-lto version:

target/debug/examples/rdr libgoblin.so | grep -A 3 syms
    dynsyms: ElfVec {
        count: 2288,
--
    syms: ElfVec {
        count: 69,
target/debug/examples/rdr libgoblin.so.lto | grep -A 3 syms
    dynsyms: ElfVec {
        count: 113,
--
    syms: ElfVec {
        count: 68,

Strangely though, a few functions were exported and I'm not sure why (i.e., goblin::elf::get_needed ends up being exported, but it's just an pub unsafe fn, but from_phdrs isn't exported...)

              cb50 _ZN6goblin3elf6strtab6impure59_$LT$impl$u20$goblin..elf..strtab..Strtab$LT$$u27$a$GT$$GT$6to_vec17hdbf1bbe1ad89f74dE (647)
            cde0 _ZN6goblin3elf6strtab6Strtab3get17h4e1ff1cea99b9c84E (879)
            d150 _ZN6goblin7archive6Member4size17h7103fd83ad14eda4E (26)
            d170 _ZN6goblin7archive6Member4trim17h734eb39b897ffff1E (98)
            d1e0 _ZN6goblin7archive6Member4name17h0906ce07ce1df4d7E (51)
            d220 _ZN6goblin7archive9NameIndex3get17h7ba57fa8352ea2feE (1007)
            d610 _ZN6goblin7archive7Archive3get17he1d4efb334604e52E (167)
            d6c0 wow_so_meta_doge_symbol (58)
            d700 _ZN6goblin5elf643dyn6impure10get_needed17hc46910f8ab7ca190E (503)
            d900 _ZN6goblin5elf643dyn11DynamicInfo3new17h555342b13899d36fE (2089)
            e400 _ZN6goblin5elf323dyn6impure10get_needed17h5bfa48fbc82bee6bE (508)
            e600 _ZN6goblin5elf323dyn11DynamicInfo3new17h86756cec6fd0af4aE (2070)

Regardless, whatever -lto is doing seems very very close to the resulting export table that we're after.

P.S., I suspect most of size that disappeared here was simply the dynamic string table massively shrinking since the symbols are no longer exported (another reason to get this resolved correctly):

du -h libgoblin.so.lto 
872K    libgoblin.so.lto
du -h libgoblin.so
1.6M    libgoblin.so

[michaelwoerister]

I just read up on this a little and -- if LLVM IR behaves like C here, which is likely -- we'll also need to mark the declarations as hidden, not just the definitions as my patch does. I'll update my branch to do that soonish. It might be a bit more complicated to implement though.

[michaelwoerister]

OK, so I pushed an update to the branch from above. Now the compiler should do the right thing when compiling with multiple codegen units.
I'll see if I can get the same principle to work cross-crate when linking statically. But we kind of lack the facilities to declare which functions should be exported and which shouldn't be. For cdylibs it's easier, there we can filter for things that are public and have extern "C" calling convention. That's not implemented in my branch yet.

[m4b]

@michaelwoerister so I'm a little confused, why aren't all pub symbols defined in the root crate exported for dylib , staticlib and rlib targets, and for cdylib, only pub extern symbols defined in the root crate exported?

It just seems to me Rust should have chosen the keyword "export", or whatever, and that was that - it would be exported. This is a very well defined notion in all 3 binary formats. It's what wasm did, etc.

[cuviper]

With monomorphization, you can easily get dynamic links even to private items. That is, some things can't be instantiated at all until their type is fully resolved, and those might still reference internal details from their original crate.

[m4b]

I don't really follow. I assume the compiler has full knowledge of every resolved symbol when it's compiling, otherwise it's an undefined reference.

Again, either the symbol is marked exported (in which case it's exported in the very defined manner in every binary format) or it isn't. I just don't see a problem here.

[cuviper]

For a silly example:

fn get_random_number() -> usize {
    4 /* guaranteed by xkcd 221 */
}
pub fn foo<T>(value: T) -> (T, usize) {
    (value, get_random_number())
}

This foo<T> can't be instantiated until we know what T is, which can be across crate boundaries. But the helper function can be created right away, and there will be a dynamic reference to it (unless you mark it #[inline]). So get_random_number is exported in the ELF sense, but not at the Rust level.

It would be very annoying if you had to manually mark every such reachable item as exported, especially when the compiler can figure this out for you.

[m4b]

So, looking at libsnappy.so some symbols from the std library are exported (but they have instantiations), which is interesting. But, taking your example from above, in this crate, call it libfoo, libfoo doesn't exist in limbo; either it's being compiled into an executable, or into another library, as a final binary target.

In both cases, all information is known, all symbols are resolved, all types have instances and their corresponding code.

If it's being compiled into an executable, and since the executable didn't define that function, it isn't exported. And that's that. (really you shouldn't export anything in executables, but whatever).

If it's being compiled into a library, and the library doesn't mark it as "exported" (or pub use, or whatever), then it also doesn't need to be exported in the ELF sense.

Yes maybe the intermediate object file "exports" it (but there isn't a dynamic symbol table in object files anyway), but that is irrelevant, since the final target doesn't export, so it isn't exported.

(and yea, I agree it would be annoying to mark every reachable item as exported, but I don't think marking which functions you want exports would require this - it is as you say, something the compiler can - and should - figure out)

[m4b]

@cuviper So, I think I see what you're getting at, but I think we have two different perspectives. Correct me if I'm wrong, but you're essentially asking what does it mean to "export" a rust generic function? And the first problem with this is that, imagining it, for example being used dynamically, it may have to expose the helper function to any consumers of the generic symbol in order for them to complete the instantiation of the generic symbol in their code? (or something like that :P)

[cuviper]

Right, I think we're almost in agreement. :) Suppose we're compiling my example as a shared library libfoo.so, and some other library or executable dynamically links to it and calls foo<Bar>. Then it will have a local instance of foo<Bar> with a dynamic reference to libfoo.so:get_random_number.

AIUI, determining that minimal set of reachable items (and hiding the rest) is exactly what @michaelwoerister is working on.

[m4b]

Yea ok, wow. 💥

[michaelwoerister]

From my point of view there are 3 concepts at play here that partially overlap:

• Syntactic visibility/access control (pub), which determines which things can be used at the Rust level.
• extern annotations, which determine the ABI/calling convention of a function.
• Symbol visibility, which only affects dynamic libraries and determines what's exported.

It's not entirely clear whether the exporting part should be inferred from the other two. In GCC you need an explicit annotation to control this: __attribute__ ((visibility ("default"))) or __attribute__ ((visibility ("hidden"))). We don't have anything like that. There are some options for inferring exporting status:

• We could say, anything that is cross-crate visible should be exported but that's more or less what we have now, and the set is too large.
• We could add the restriction that only cross-crate visible things with C calling convention are exported from cdylibs. But what about other calling conventions that are also used from C?
• We could say that only cross-crate visible things with a calling convention not specific to Rust should be exported from cdylibs, but that seems a bit arbitrary.
• We could say that only cross-crate visible things with the #[no_mangle] attribute get exported from cdylibs. That makes sense because anything else will have an unpredictable symbol name. But it's a bit non-obvious maybe.

That last option doesn't sound too bad to me but one might still want to explicitly control what's exported and what isn't. A #[export] attribute (that would also map to dllexport on Windows) would do the trick. But in Rust dylibs things are a bit different again, there you'd probably want anything exported transparently -- but Rust dylibs are a strange thing anyway because of generics.

[alexcrichton]

To recap what the intentions have been in the past:

• In dylibs everything reachable is exported, for reasons that @cuviper listed above.
• In staticlibs, executables, and cdylibs, the intention is that only non-Rust ABI items with #[no_mangle] are exported, everything else is a private detail.

That is, we've attempted to infer from syntactic visibility and extern what to export. We may have latent bugs, however, which need to be fixed to achieve these goals. So far at least I haven't seen a use case which doesn't fall into those categories. Having an explicit #[export] would be nice from time to time, but I've often felt that it'd be best to see how far we can get without it before we reach for that.

[michaelwoerister]

OK, I can see that there are a few bugs in the implementation then. I'll update my branch to fix those.

[michaelwoerister]

I've pushed a new version of the compiler to https://github.com/michaelwoerister/rust/tree/symbol-visibility. This version should now do the right thing for cdylibs (i.e. hide anything that isn't marked with #[no_mangle]). It passes all existing tests but I we'll also need some new ones to make sure we don't export too much for cdylibs, etc. Also, I've not tested on anything but Linux so far.

[m4b]

@michaelwoerister

Unfortunately this doesn't seem to have any effect for me, unless I'm not doing it correctly. How are you testing/checking for whether the symbols are exported?

Might I suggest we agree upon either:

1. objdump -T
2. nm -D
3. Or, my personal favorite, cargo sym -e 😎

For a cdylib with a single exported symbol (ie no_mangle, extern C fn), I would expect no more than 20 symbols (being conservative, really there should be 1 but whatever), not the 2k we're seeing now.

After building, I basically copy all the std stage two artifacts to the stage 2 rustc build directory, copy the tools, create a simple file with an exported symbol and then do:

LD_LIBRARY_PATH=. ./rustc -L. --crate-type=cdylib lib.rs

And check the resulting exports, which is still 2k+

Are you seeing different results (with one of the above method for checking exports)?

[m4b]

So executables on x86 with unpatched compiler seem fine actually (likely because default for executable is everything is private). This includes no unnecessary plt relocations.

Been messing with arm and aarch64 and they also seem fine with unpatched compiler.

[michaelwoerister]

Well, "just" fixing cdylibs and staticlibs is good too :)

I have a pretty cleaned up version of my fixes here:
https://github.com/michaelwoerister/rust/commits/hidden-symbols

Just needs a test case or two, then I'll make a PR.

[m4b]

I think this is resolved for cdylibs so closing, feel free to reopen if desired for some reason.

[lilianmoraru]

I think this is even more relevant now with pub(crate) - I see that currently the visibility is global.

[fzyzcjy]

Hi, seems that I still face the problem. I use crate-type = ["cdylib"], but the generated .so file (built for android) has tons of symbols in addition to the no-mangle extern C functions (which are the only ones that should be seen). For example, 0006eff4 t _ZN15vision_utils_rs3api11debug_throw17h4962635ff6e9c7bdE which is a pure Rust function in my own code. Another example, 002f3f3e t _ZN47_$LT$std..fs..File$u20$as$u20$std..io..Read$GT$4read17h72d3dd8a177ea43cE which is a std thing.

It is cdylib, and the users of this .so will only use the nomangle extern C fn. So all other symbols should not be present.

Thanks!

[bjorn3]

Lower case types like t in this example indicate local symbols. Upper case types like T indicate exported symbols. _ZN15vision_utils_rs3api11debug_throw17h4962635ff6e9c7bdE and _ZN47_$LT$std..fs..File$u20$as$u20$std..io..Read$GT$4read17h72d3dd8a177ea43cE are likely used directly or indirectly by an exported function. To get rid of local symbols you need to strip the library/executable.

[fzyzcjy]

@bjorn3 Thank you! Let me have an experiment.

[fzyzcjy]

@bjorn3 Yes, it does work. Thanks again!

[dcow]

@bjorn3 Yes, it does work. Thanks again!

@fzyzcjy it would be nice if you included your solution for posterity.

[fzyzcjy]

@dcow Sure. Simply call strip myfile.so and everything is done.

(Indeed originally I called that, but not sure why the pipeline change later omits that strip call. - but that is unrelated to this question)

[knggk]

Hello,

There is one case where having all symbols being dynamic makes sense for cdylib. Some C programs use libc's backtrace_symbol() to generate a backtrace. However backtrace_symbol only reads dynamic (global) symbols and not local ones, in an attempt to contain the ELF handling logic in libc (ref: https://bugzilla.redhat.com/show_bug.cgi?id=169017#c1).

So this present issue effectively removes backtrace_symbol()'s ability to provide symbols for frames in the Rust .so file (the cdylib). As an alternative, I tried to build the crate as dylib, which keeps all symbols dynamic. However it then fails to be loaded from C (mangling? specific linker options? my Rust .so itself depends on a C .so).

Question: is there an option for cdylib to preserve the old behavior, having all symbols dumped in the dynamic symbol table?

[bjorn3]

Even with the old behavior we didn't include all symbols in the dynamic symbol table. Many have to be made private to prevent symbol conflicts. That includes all #[inline] symbols and all generic functions as those are duplicated into every codegen unit that uses them.

In any case backtrace_symbols isn't all that reliable even if all symbols are in the dynamic symbol table. It requires frame pointers to be preserved (not preserved by default on most targets for rustc) rather than walking the stack using the information stored in the .eh_frame section. It ignores inlining information from the debuginfo (only present if you enable debuginfo in the first place, but still).

Maybe you could export a function from the rust cdylib doing println!("{:?}", std::backtrace::Backtrace::capture());? The rust cdylib almost definitively has the backtrace generation code in std::backtrace anyway for printing backtraces on panics. Might as well use it in your C program if you have control over it.

[knggk]

That's an interesting take. Will try it out to see how that could work. Thank you @bjorn3 !

[bridiver]

This problem still exists if you link rust object files into a macos framework. It seems related to #73295 and this is also for chromium, but the problem I'm seeing is that all rust symbols are exported in the framework and we don't want any of them to be (we only want to export the symbols for the framework methods we are making available).

[danakj]

@bridiver I believe this is #73958, cxxbridge #[no_mangle] symbols, which exist for the purpose of ffi, get implicitly marked exported as well.

[bridiver]

@danakj cxx symbols are usually the first ones that show up in the error list (and then truncated there due to the number of errors), but there are tens of thousands of them including rust core lib functions that all get exported. The static library we used to generate from cargo had tens of thousands at least. See which looks like most of core and std, e.g.: in the first post here

[danakj]

Yeah, ok you're right, we're seeing different things on Windows vs Mac.

On Windows, it is just the #[no_mangle] symbols being exported. That is #73958.
On Mac/Linux it is all Rust symbols being exported. That is this issue?

What I don't understand is why Windows doesn't end up with all the Rust symbols exported but Linux/Mac do.

[bridiver]

On Windows, it is just the #[no_mangle] symbols being exported. That is #73958. On Mac/Linux it is all Rust symbols being exported. That is this issue?
What I don't understand is why Windows doesn't end up with all the Rust symbols exported but Linux/Mac do.

I believe that's what this issue is, but this was mainly targeted at dynamic libs. It doesn't seem like a major issue for Linux because they are presumably removed when stripping the exe? The problem is with something like macos framework (or other dynamic lib) that needs to export some symbols, but doesn't want to export all symbols