Debian Packages Strip Symbols from Shared Object Files

[brianrob]

The process of producing the .deb package files strips symbols out of the .so files. However, we explicitly include function-level symbols in native binaries for use during performance tracing. tar.gz files are unaffected.

Steps to reproduce

1. Install the .deb files for the shared framework for any of the Ubuntu distros either directly or via apt-get.
2. Run nm </path/to/libcoreclr.so>

Expected behavior

Function-level symbols are displayed.

Actual behavior

No symbols are displayed.

Environment data

[gkhanna79]

@chcosta Will this be addressed as part of your symbol packaging work?

CC @ericstj @noahfalk

[chcosta]

This is not directly related to the symbol packaging work I'm doing. @dagood has more experience with the symbol stripping stuff.

[dagood]

As written, this has more to do with how deb-tool works. That kind of step is where symbol stripping is commonly done in Linux package manager flows (so I hear). I don't know where deb-tool comes from. It seems unrelated to packaging work to me, unless the root cause is how Core-Setup gets CoreCLR bits rather than deb-tool like the description says.

[omajid]

I am from Fedora land, where a similar problem would happen, I suppose. From what I understand, the packaging tool, during the package build process, goes over all binaries and strips them. The stripped binaries are put into a separate package (-dbgsym for Debian, -debuginfo for Fedora).

See AutomaticDebugPackages for how debian packages get stripped during build.

The debian announcement lists a few options to disable the stripping during package build.

[brianrob]

My understanding matches that of @omajid. Likely we just need to disable the stripping when producing packages.

[omajid]

A tangential question: is there some way to automatically verify that symbols needed for performance testing are available?

[brianrob]

Yes, you could do something like nm libcoreclr.so and make sure that the result is not "no symbols".

[brianrob]

Or, you could also take the output of nm before and after the production of the package and diff them.

[omajid]

Thanks, those are good suggestions.

That said, is there some way to verify the functionality itself? What I mean is that the format for debugging symbols seems to be evolving. I know that (some versions of) lldb have trouble reading debugging symbols processed by some versions of rpmbuild. Is there a way to verify that the symbols present are sufficient for the performance tracing implementation in .NET Core to work?

[goldshtn]

I think it would be perfectly fine to provide a separate debuginfo package and put full symbols in it. This makes installation faster and smaller, and most users and production deployments won't need the debuginfo anyway. I realize it might complicate things on your end a bit, but that's what most packages use. Even fundamentals like libc would usually have minimal symbols embedded and then a debuginfo package with full symbols.

@omajid Performance tracing actually works just fine without symbols. You simply don't have symbols on your call stacks for libcoreclr.so and other native modules. It is not critical for performance tracing user code, although is of course very nice to have. There is an even bigger problem around tracing crossgen'd symbols, which I brought up with Brian on a DL.

[omajid]

I think it would be perfectly fine to provide a separate debuginfo package and put full symbols in it

This is what we are doing in Fedora [1]. There's a .NET Core package (dotnetcore) and a debug symbols package (dotnetcore-debuginfo). If debug symbols aren't required for general functionality, I suggest that the debian packages follow the same format. But see also https://github.com/dotnet/core-setup/issues/1599 where packaging names and other issues are being discussed.

[danmoseley]

Assigning to @rakeshsinghranchi per @gkhanna79
@rakeshsinghranchi please let us k now if this will not be resolved by 5/10 ZBB cutoff for 2.0.

[eerhardt]

@chcosta - is this the same thing you are working on? Same as https://github.com/dotnet/core-setup/issues/186?

I'll assign to you.

[chcosta]

@brianrob , can you help me understand this change? I actually think this is the opposite of the thing I recently worked on. #2450 was a change to ensure we stripped symbols from our nupkgs and produced separate symbols packages. It sounds like you're asking for the opposite behavior for deb packages. @rakeshsinghranchi , is this something you're aware of / familiar with?

[rakeshsinghranchi]

Moving to 2.1.0 per response from @brianrob . His suggestion is to get the fix in 2.1.0 sooner.

[rakeshsinghranchi]

@dagood ,is dotnet/source-build#272 required to be ported for this issue ?

[dagood]

No, porting that (removing -strip-symbols) to Core-Setup official builds would make it impossible to create symbol packages, which we use to submit the stripped-out native symbols to the symbol server. In source-build, the output isn't put on a symbol server, so disabling symbol stripping is ok.

That said, I don't know the proper fix for this issue.

[omajid]

That said, I don't know the proper fix for this issue.

Just thinking out loud here. removing -strip-symbols leaves the symbols in-place. If they were left in place, we could then create -dbgsym packages from them. That would solve that problem. The other issue, of having symbols for symbol server can be fixed by adding some feature somewhere in the pipeline to create separate symbol files.

Maybe we can add an option to the build for this.

• default build leaves symbols in native libraries
• -separate-symbols would leave symbols in native libraries, but also create separate native symbol files that could be uploaded to the symbol server
• -strip-symbols can be kept as it is now, except it wont be used by many (not by official builds, not by source-build)

[dagood]

That's a good point, I don't see a reason stripping has to happen during the initial native build. The flag just runs commands on the outputs:

https://github.com/dotnet/core-setup/blob/b5dd381dd3a586640e4bfe4804d6563861eaab20/src/settings.cmake#L84-L107

(-separate-symbols should also output the stripped native library files. I believe we still want to use them in some scenarios to reduce file size.)

[rakeshsinghranchi]

@dagood , can you help with enabling --separate-symbols option , am not familiar with build process. Thanks.

[dagood]

I can't right now (and my cmake knowledge is pretty rudimentary), but the overall idea, I think, is to copy strip_source_file so we don't "lose" the version with all the symbols. Then we get all three files from a build: so with symbols, so with symbols removed, and so.dbg that contains the stripped symbols. Then they can be put into the appropriate artifacts.

[rakeshsinghranchi]

I don't think it can be done for release 2.1.0 . Moving this to future.

[dagood]

This came up recently in dotnet/symstore#126 due to the unexpected difference between the libcoreclr.so shipped in our Debian package vs. other artifacts. Looking over this issue knowing a little more about how the problem comes up, I think it got off track.

I don't think turning off symbol stripping and leaving it to debtool helps. Symbol stripping is not all-or-nothing: debtool strips more than it should for .NET Core. We need to figure out how to make debtool leave the function-level symbols in the so, no matter what else we do, so that perf tracing works with this installation method. (Keeping parity with other artifacts.)

We do have this issue tracking making proper debuginfo/dbgpkg packages now, though: dotnet/core-setup#8356. 🙂

[omajid]

I don't think turning off symbol stripping and leaving it to debtool helps. Symbol stripping is not all-or-nothing: debtool strips more than it should for .NET Core. We need to figure out how to make debtool leave the function-level symbols in the so, no matter what else we do, so that perf tracing works with this installation method. (Keeping parity with other artifacts.)

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=595810 has some hints on how to do that. I haven't tested/verified it.

[NikolaMilosavljevic]

@brianrob is this still an issue for perf tracing or do you utilize a workaround? Is this needed for 5.0?

[brianrob]

@NikolaMilosavljevic I think its safe to close this now. We have a tool for downloading symbols now. Thanks.