Allow loading of llvm plugins on nightly

[ZuseZ4]

Based on a discussion in #82734 / with @wsmoses.
Mainly moves this behind a -Z flag, so it can only be used on nightly,
as requested by @nagisa in #82734 (comment)

This change allows loading of llvm plugins like Enzyme.
Right now it also requires a shared library LLVM build of rustc for symbol resolution.

// test.rs
extern { fn __enzyme_autodiff(_: usize, ...) -> f64; }

fn square(x : f64) -> f64 {
   return x * x;
}

fn main() {
   unsafe {
      println!("Hello, world {} {}!", square(3.0), __enzyme_autodiff(square as usize, 3.0));
   }
}

./rustc test.rs -Z llvm-plugins="./LLVMEnzyme-12.so" -C passes="enzyme"
./test
Hello, world 9 6!

I will try to figure out how to simplify the usage and get this into stable in a later iteration,
but having this on nightly will already help testing further steps.

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @varkor (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[varkor]

r? @nagisa

[ZuseZ4]

I think I pushed the second commit from the wrong build dir, but now it should be all right, unless you have some further change requests?
I will look on how we can expose the missing symbols by default so we don't need the shared lib llvm build anymore, but this will be another PR.

[nagisa]

nit: tracing already includes information about functions and modules that the debug call is in. This should perhaps say more about what's being done here precisely:

Suggested change

	Ok(_) => debug!("configure_llvm: {}", plugin),
	Ok(_) => debug!("LLVM plugin loaded {} ({})", path.display(), plugin),

Or something along those lines.

[ZuseZ4]

Makes sense, I will update it in this case. But how about this section, is it outdated? https://rustc-dev-guide.rust-lang.org/compiler-debugging.html#logging-etiquette-and-conventions

A loosely followed convention is to use debug!("foo(...)") at the start of a function foo and debug!("foo: ...") within the function.

[nagisa]

Huh, I wonder if that convention predates the use of tracing for logging. @oli-obk do you know?

[nagisa]

@bors r+ Nice!

[bors]

📌 Commit abdd24a has been approved by nagisa

[bors]

⌛ Testing commit abdd24a with merge a1411de...

[bors]

☀️ Test successful - checks-actions
Approved by: nagisa
Pushing a1411de to master...

[bgat]

I'm trying to load and use my own LLVM plugin from rustc, and getting the following error, even though the very same .so file works when use it via opt:

error: failed to run LLVM passes: Failed to load pass plugin/home/ubuntu/coast/projects/build/DWC/libDWC.so

I'm using the legacy plugin format. Is the above related to the, "Right now it also requires a shared library LLVM build of rustc for symbol resolution" requirement?

[ZuseZ4]

what do you get from opt --version ?

I don't think that the latest LLVM (which rustc is building or downloading by default) does support legacy plugins at all.
I was actually about to propose to remove this legacy fallback one of these days.
Is there a reason why you can't move to the newPM?

[bgat]

The plugin I'm using right now was written as a legacy plugin. I was hoping it would Just Work for rustc. :-)

ubuntu@ip-172-31-49-79:~$ opt-15 --version
Ubuntu LLVM version 15.0.6
Optimized build.
Default target: x86_64-pc-linux-gnu
Host CPU: skylake-avx512

[ZuseZ4]

I think in this case you should probably just take an older rust commit to build. I don't think that you will get a legacy commit working eith an up do date LLVM

[bgat]

I'll update my plugin to the modern format, and try again. I don't think I can go far enough backwards with rustc to make a difference here. Will let you know.

[CXWorks]

Hi @ZuseZ4 , I'm trying to use my own afl-pass with asan like below:

RUSTFLAGS=" -Z llvm-plugins=./afl-llvm-pass.so -C passes=hello" cargo build --bin xxx

But it seems doesn't work: error: failed to run LLVM passes: unknown pass name 'hello'

[ZuseZ4]

@CXWorks You should look at newer PRs mentioning LLVM plugins - My original plugin infra iirc got removed and replaced by a newer, since LLVM plugins for the new PassManager work a bit different. You would need to look at those PRs to figure out how to use them. For my own project I moved away from the plugin interface, since it was too limited, so I didn't follow closely.

[CXWorks]

Thanks for your quick response!