Make "-fno-inline" compile option more accessible

[prakashsurya]

When functions are inlined, it can make the system much more difficult
to instrument using tools such as ftrace, BPF, crash, etc. Thus, to aid
development and increase the system's observability, when the
"--enable-debug" flag is specified, the "-fno-inline" compilation option
will be used for both userspace and kernel modules.

[prakashsurya]

This is a first attempt at implementing a solution for #6604. I've done minimal verification of this patch in my local VM running Ubuntu 16.04; see below...

When using this patch and building with --enable-debug:

$ sudo /usr/share/bcc/tools/trace zil_lwb_commit
...
32740  32740  fio          zil_lwb_commit
579    579    fio          zil_lwb_commit
579    579    fio          zil_lwb_commit
579    579    fio          zil_lwb_commit
579    579    fio          zil_lwb_commit
...

When using this patch and building with --disable-debug:

$ sudo /usr/share/bcc/tools/trace zil_lwb_commit
Failed to attach BPF to kprobe

When using -finline, as is the case with --disable-debug, the zil_lwb_commit function will be inlined, which means it cannot be instrumented using BCC's trace tool. When using -fno-inline, this function will be available to be traced, due to it not being inlined by the compiler.

[behlendorf]

I'm good with this. Can you please rebase the PR on master and drop the Configure-zfs: line so we can get a clean test run too. The build failures were expected and caused by overriding the --with-spec=redhat configure option which is used by the CentOS TEST builders.

[prakashsurya]

done.

would it be reasonable to add --enable-debuginfo to one of the existing tests? and/or add a new test for this? the issue of this option increasing stack usage, and potentially causing stack overflow problems, worries me.

[behlendorf]

Definitely, we could probably make it the default for all the TEST builders. This would both help maximize the debugging we can get out of the builders if they ASSERT/panic and make it more likely to catch stack issues. My only concern is that gcc still inline functions we've annotated that must be force inlined. For example, the ZFS Channel Programs code requires this to avoid trashing the stack.

[prakashsurya]

My only concern is that gcc still inline functions we've annotated that must be force inlined.

to make sure I understand correctly, the concern is that these functions might not get inlined? when, in fact, we need them to be regardless of whether or not -fno-inline is specified? I'm OK blocking this until I can verify that the annotations still work even when --enable-debuginfo is used.

[behlendorf]

the concern is that these functions might not get inlined?

Yes, exactly. There are known cases where we will overrun an 8K stack if certain functions aren't inlined. The good news in that for the last few years all x86_64 kernels are all built with 16K stacks by default so things are moving the right direction and we have some elbow room. The bad news is that 32-bit kernels and 3.14 and older kernels still default to 8k stacks.

[behlendorf]

I've opened openzfs/zfs-buildbot#90 to pass --enable-debuginfo by default. Once we're sure that doesn't expose any new issues we can run this PR through testing again to see if it causes a significant increase in stack depth.

[behlendorf]

This looks good. It doesn't appear to have made things worse, I'm running a few builds through again to make sure but if everything still looks good tomorrow I'll merge it.